AirdropletResearch desk
Menu
← Back to video archive

Airdroplet AI summary

Create AMAZING 3D Scenes in Blender with ComfyUI! (Tutorial)

May 15, 2025Matthew BermanAI score 10028,547 views

Watch original on YouTube ↗

AI-generated summary

Okay, so imagine taking the power of AI image generation and bringing it right into your 3D modeling software. That's exactly what this video is about! It shows you how to set up and use an NVIDIA-backed project that integrates ComfyUI (a popular node-based AI workflow tool) directly into Blender (the free 3D software) to create amazing 3D scenes guided by text prompts. It's a pretty technical setup, requires some serious hardware, but lets you locally generate and change the look of 3D environments using AI.

Here are the key things covered:

  • What this project is: It's called the "NVIDIA RTX 3D Guided Generative AI Blueprint." Think of it as a starting point or reference sample provided by NVIDIA. Its main goal is to let creators guide the design of simple 3D objects and scenes in Blender using AI image generation driven by prompts.
  • It's ComfyUI inside Blender: The magic happens by having ComfyUI running within the Blender interface. This means you're using ComfyUI's flexible workflow specifically for altering your 3D scenes.
  • Core Functionality: You can take a basic 3D scene built in Blender and "apply any skin to it" using AI. It doesn't just paint over it; it can understand the scene and intelligently fill in blank areas or elaborate on the existing geometry based on your prompts.
  • Open Source, Free, and Local: A major highlight is that this entire blueprint is completely open source, free to use, and runs entirely locally on your own computer once everything is downloaded. You don't need an internet connection to run the generations after the initial setup.
  • NVIDIA's Goal: NVIDIA partnered on this video for two main reasons: one, to show how to set it up with their NIM (NVIDIA Inference Microservice) framework, and two, they are really looking for feedback from users. They want people to try it, build on it (the code is available), and tell them what they think about these open-source projects.
  • Demonstrated Capabilities: The video shows examples like transforming a simple 3D village scene. Starting with a "quaint village in the mountains, in winter," it's changed to a "hyper modern town" and then a "cartoon style" town just by changing the text prompt.
  • Adding Objects: You can add new 3D assets (like a boat) into the Blender scene. The AI then tries to integrate it based on the prompt.
  • Changing Perspective: Since the output is 3D, you can rotate the scene in Blender and run the generation again from a different camera angle to get a consistent look across views. The presenter shows generating the boat scene from a top-down view. The possibilities feel "truly endless" with this ability.
  • Significant Hardware Requirements: Here's the big hurdle for many: you need some serious hardware to run this right now. The minimum requirements listed are specific high-end NVIDIA RTX GPUs (like the 5090, 5080, 4090, 4080, 4090 laptop, or the RTX 6000, which the presenter uses) and a substantial 48 gigabytes of system RAM. The presenter acknowledges these are "pretty substantial." Good news is they are working on making it more efficient so more GPUs will be supported in the future.
  • The Installation Process (It's a bit involved): Getting this set up isn't the easiest thing, but the video walks through the steps:
    • Download and run the NVIDIA NIM prerequisite installer. This sets up a lot automatically, including NVIDIA AI Workbench.
    • Install Git (version control) using PowerShell or a similar command line tool.
    • Install Microsoft Visual C++ using a similar command line method.
    • Download and install Blender (it's free).
    • Get a Hugging Face access token. This is needed because the project uses open-source models like the Flux diffusion model, which you download via Hugging Face. You only need read access. You also must go to the Hugging Face pages for all the listed required models and accept their terms of service.
    • Clone the code repository (download the project code) using Git.
    • Navigate to the downloaded folder and run the setup.bat file. This is a crucial step that downloads all the AI models locally, which can take a while. It provides paths you'll need later.
    • Open Blender, go to Edit > Preferences > Add-ons, find and enable the "ComfyUI Blender AI node." Paste the ComfyUI and Python paths from the setup.bat output into the add-on settings.
    • Finally, open the specific provided Blender file (guided gen AI BP.blend) which contains the setup.
  • Using the Interface and Workflow: Once the .blend file is open, you see the familiar Blender interface with the ComfyUI nodes visible at the bottom.
    • Opening the Blender System Console (Window > Toggle System Console) is recommended because it lets you see the AI process running in the background, which the presenter finds helpful and reassuring.
    • Before running any generations, you need to click the "Launch Connect to Comfy UI" button within the Blender add-on. This spins up the local AI server and connects Blender to it, which takes time, especially the very first time you do it.
    • You can drag included 3D assets into the scene from a panel.
    • The primary way to control the AI output is by editing the text prompt in a specific panel. The presenter suggests copying the prompt to an external text editor for easier editing.
  • Crucial Insight: The Prompt is King: A key learning point is how strongly the AI model adheres to the prompt. If you add a 3D object asset to the scene but your text prompt doesn't mention it or contradicts it, the AI will try to interpret that object based on the prompt. The example of adding a boat asset to a scene prompted as a "village" resulted in the boat looking like a weird boat-shaped fountain because the prompt didn't mention a boat or water. You must explicitly update the prompt (e.g., "and there's a yacht in the middle of the town on water") if you want added objects to be rendered as intended and fit logically into the AI's generated scene. This was a surprising and important detail.
  • Presenter's Experience: Even as someone new to Blender, the presenter found the core usage straightforward, though the initial setup was challenging. He feels that now that he's done it, it's much simpler the next time.
  • Call to Action: NVIDIA is actively soliciting feedback on this blueprint. If you try it (and have the hardware!), share your thoughts and creations, and consider contributing to the open-source project.

In short, this is a powerful but demanding way to integrate AI generative capabilities directly into your 3D workflow using Blender and ComfyUI, driven by an NVIDIA blueprint that runs completely locally. Setting it up is technical, and the hardware barrier is high currently, but the ability to creatively guide 3D scene design with text prompts is a cool step forward.

Video transcript

Open transcript
This is Comfy UI inside of Blender, which allows me to do really cool things like take this 3D environment and apply any skin to it. Not only that, it'll take liberties and actually fill out any of the blank spaces in the 3D environment. All of this is controlled by Comfy UI right there. And I'm going to show you how to do it. This is the NVIDIA RTX 3D Guided Generative AI Blueprint. It's a ready to go reference sample that people can use as a starting point for things to build using NIMh microservices. This is completely open source, completely free, and running completely locally. And with this specific blueprint, creators can use simple 3D objects laid out in Blender and guide the design of it using simple image generation with prompts. NVIDIA partnered with me on this video for two reasons. One, they wanted me to make a tutorial showing you how to set this up with NVIDIA NIMh microservices. And to be frank, it's not easy, but I'm going to show you step by step. And number two, NVIDIA is looking for your feedback. They want to hear what you think about these open source projects. They want you to build on top of it. They want you to fork the repo. I'll drop all of the links down below so you can do that. But yeah, most of all, they want to know what you think. So let me show you a couple of the things that you can do with this, and then I'm going to show you how to install it. So we have the first scene, which is a 3D representation of kind of a little town with a fountain in the middle. And here is the prompt. So I just copied it to notepad to make it a little bit easier to read. A stunning professional photo of a quaint village in the mountains. In winter, a snow covered fountain is visible in the center of the town square. And that's what we're seeing here. But what if we wanted to change that to a hyper modern town? Let's see what that looks like. All right, there it is. Hyper modern version. What if we want it in cartoon style? There it is. All right. And last, let's just put a boat right in the middle. So I just placed the boat asset right here and changed the prompt slightly. There it is. And remember, this is 3D. So we can just simply rotate the environment and get another angle. Let me show you. Okay, here's one from the top down. Let's hit run and see it again. All right, there it is. So the possibilities are truly endless. Let me show you how to install this. Now, all of the code and the instructions are on this GitHub page, which I'm showing you now. And I'll drop a link to this in the description below. The minimum requirements to run this are pretty substantial. Let me show you. You need one of these GPUs. So an RTX 5090, 5080, 4090, 4090 laptop, 4080 or RTX 6000. That's what I'm running it on. And 48 gigabytes of system RAM. So if you have that, you can get this up and running. And keep in mind, they're working on making this entire flow more efficient. So more GPUs will be supported soon. So the first thing you're going to do is scroll down a little bit and look for this link right here. Download the NIMM prerequisite installer. It's going to download the zip file, which basically sets the majority of things up for you automatically. Once you do that, you're going to walk through the installation. So NIMM setup, you're going to hit continue, agree and continue. And it's going to download the AI Workbench, install it. I already have it downloaded. So it might be faster than normal. And you're done. Go ahead and click close. Now, the next thing you're going to need is get and that's version control. So you're going to open up PowerShell. You're going to copy this command right here. When get installed dash dash ID, get dot get and paste it in hit enter, which will download it and install it for you. Now it already found a version. I already have it and it's trying to update it instead. All right. Once you see this successfully installed, it's done. Next, you're going to need to install Microsoft Visual C plus plus. And the command is right here. Same page as before. So when get installed, Microsoft dot VC redist dot 2015 plus dot X64, just go ahead and copy it, switch back to your PowerShell, paste it in hit enter. So for me, I already installed it. It's not going to try to install it again, but for you, it'll probably take its time if you don't already have it installed. Next, you're going to need Blender. So you can download Blender from this link right here or just Google Blender. It is free. Blender is super simple to install. Just go through the install process and that's it. So the next thing you're going to need is a hugging face key. And the reason you need that is because this entire flow from Comfy UI into Blender uses the flux model, which is a diffusion model, really popular open source, very easy to use, but you do have to download that and you do so through hugging face. So log into hugging face, click this little icon in the top, right? You're going to hit access tokens right there. Then you're going to click this create new token button right there. You're going to select the read. You only need read. You don't need to overcomplicate it. I just name it NVIDIA Comfy and create tokens. So then just copy your token and have it ready. Now, one thing you're going to need to do is accept the terms of service on each of these models. There's a list of models on the instructions page. All of these, you're going to need all of them. All right, next, you're going to clone the repo. That is actually this entire framework. So right here, get clone NVIDIA AI blueprint, 3D guided gen AI RTX. Go ahead, click copy. And so wherever you want this located, go ahead and paste it, hit enter, and it's going to download it to that folder. Next, you're going to CD into that new folder that you just downloaded. All right, now we're in there. Then navigate to that folder within your finder window. Look for setup.bat, double click it and let it run. This will probably take a while because it is downloading a ton of data to your local computer. This is downloading the models so you can have them locally. All right, when it's done, you're going to get this output, ComfyUIPath and PythonPath. And so don't close that window, keep it open because you're going to need both of those things. Next, open Blender. Once you have Blender open, you're going to go to the edit dropdown and then preferences. Then you're going to click add-ons and then you're going to look for this ComfyUI Blender AI node right here. Make sure it's checked, scroll down, and here is where you're going to paste the ComfyUIPath and the Python path from that previous screen. So I already did it. I have it right here. I'm not going to touch it. And that's it. Then the last thing you're going to need to do is simply open the blueprint that they provided. So we're going to go to file, open, then within documents slash blender, you might have to look around for this, but that's where it put it for me. You're going to look for this guided gen AI BP dot blend file and go ahead and click open. And that's it. And it'll take a few seconds, hopefully not too long. And we have it open right here. So let me tell you something. I had never used Blender before. I am a complete novice and I am learning, but it's pretty straightforward and there's a ton of tutorials online. You can also just ask AI, of course. So what you're seeing down here, this is Comfy UI. Everything else is Blender. Now, one thing I like to do is go up to this window dropdown, hit toggle system console, and we can actually see everything going on in the background. And that's nice because it gives me confidence that I actually see things progressing. Now, before you do anything else, what I like to do is hit this red button here, launch connect to Comfy UI. And when you do that, it's going to take a little while, especially the first time you do it. And what this is doing is it's spinning up the NIM, making sure that Comfy UI and Blender are both connected to it. And this all happens before you run your first generation. Now, while that's going, let me show you a couple of things. Down here is where you'll see the different assets that you can play around with. So you can simply grab one and drag it in there like so. And I just heard a little beep, which means that the connection to the Comfy UI server is ready. So here's pending and running zero of zero, because we have none running. I added a nice second fountain there. And before I do that, I'm going to show you the rest. So down here, you can zoom in, you can scroll around. The prompt that you're going to be adjusting is found right here in this 001 panel. As I said before, what I like to do is copy it, go over to a text editor, paste it in and edit the prompt from here. You can make the prompt anything you want, and it will reflect in the output. Now, a couple of things to keep in mind, the model really likes to adhere to the prompt. And so if I simply dragged a boat right here and it didn't make sense in the scene, it's not going to look like a boat. It's going to look like a fountain shaped like a boat. So you would explicitly want to say, and there's a yacht in the middle of the town on water. And if you don't, I'll show you what happens. So there's that boat that looks like a weird fountain in the middle of the town. So if you want it to be a boat, make sure you say boat in the prompt. So there we go. I fixed it up and now you see it's an actual yacht. So all of this is completely open source, completely free, completely local. Once you download everything, you can run it locally with no internet. And I want to give a special thanks to the team at Nvidia who worked with me, showed me how to do all of this. It wasn't easy to get going the first time, but honestly, now that I've done it a couple of times, it's pretty straightforward. Hopefully this tutorial helps you. And again, Nvidia is looking for you to provide them with feedback on what you think about this and their other blueprints. Drop your thoughts below in the comments. Let me know what you think. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.