1 00:00:02,159 --> 00:00:08,580 Hey, we are live. Welcome to Freedom Tech Weekend. My name is Marks. I am your host 2 00:00:08,960 --> 00:00:15,720 week in and week out. It's just me and the chat. What's up, chat? So go ahead and toss 3 00:00:15,960 --> 00:00:22,640 in any questions that you have. Today we are talking about open source LLMs, how you can 4 00:00:22,660 --> 00:00:27,980 run your own AI at home on your own laptop. You can probably do it on your phone too. 5 00:00:28,140 --> 00:00:36,200 Today we're not going to talk about the phone, but I'm going to show you some tools, and then you can do your own exploration from there and maybe even get it going on your phone. 6 00:00:37,020 --> 00:00:38,840 But this is Freedom Tech Weekend. 7 00:00:39,020 --> 00:00:39,560 What is this? 8 00:00:39,700 --> 00:00:46,000 This is something that we do every week where we show you one thing that you can play with this weekend when you have free time. 9 00:00:46,400 --> 00:00:54,820 One open source tool or Freedom Tech tool that you can use to start decoupling your life a little bit more from closed technology. 10 00:00:55,820 --> 00:00:58,920 ways that you can have more control over your data, more control of your workflow, 11 00:00:59,160 --> 00:01:04,019 just more freedom and independence in the way that you use computers. Because we, 12 00:01:05,180 --> 00:01:11,340 over the last 20 years or so, we have coupled our technology with other people's servers, 13 00:01:11,640 --> 00:01:16,020 other people's computers. We call it the cloud. And there's a lot of good reasons for this. 14 00:01:16,420 --> 00:01:20,420 We were able to scale. We were able to do so many more things, communicate so much better 15 00:01:20,480 --> 00:01:25,080 over the internet. But this has come with some trade-offs. And the biggest of that is our privacy 16 00:01:25,080 --> 00:01:32,160 and access to our data. You see data leaks all the time. You see your data being sold to 17 00:01:32,360 --> 00:01:38,020 advertisers, to third parties, to governments. And so we need to look at these trade-offs and 18 00:01:38,240 --> 00:01:43,760 let's see, is it worth sharing our data with third parties if that's what we're getting in return? 19 00:01:44,619 --> 00:01:49,180 And then you can make your own decisions. There are still plenty of cloud-based things that I use 20 00:01:49,180 --> 00:01:55,400 every day and I make that trade off. But there are plenty where I have decided to take back a 21 00:01:55,480 --> 00:02:00,740 little bit and do it on my own or use other tools that help me do it in a more private way. 22 00:02:01,280 --> 00:02:06,180 If you don't know who I am, again, my name is Marks and I run a software startup, 23 00:02:07,140 --> 00:02:14,959 an AI startup called Maple AI. Maple lets you run some of the most powerful, I think we have 24 00:02:14,960 --> 00:02:19,120 the most powerful open source models right now, and they're fully encrypted. So you get to run 25 00:02:19,300 --> 00:02:24,840 confidential AI in the cloud, but end-to-end encrypted with your own private encryption key 26 00:02:24,900 --> 00:02:29,080 that is generated in a secure enclave on the server. Today, I'm going to show you how to run 27 00:02:29,180 --> 00:02:33,920 local AI. So you might say that this is a competitor to what we offer, but really it's 28 00:02:35,020 --> 00:02:40,000 all part of the same package, right? We want people running local AI, and then we want people 29 00:02:40,200 --> 00:02:44,940 using Maple when they need something more powerful, because your laptop is simply just not as 30 00:02:44,940 --> 00:02:47,220 as the GPUs that we have in the cloud. 31 00:02:47,890 --> 00:02:51,020 So we're going to show you kind of the whole package and help you out. 32 00:02:51,880 --> 00:02:53,140 All right, we've got people joining. 33 00:02:53,400 --> 00:02:54,040 Welcome, everybody. 34 00:02:54,250 --> 00:02:55,500 I see people hopping in the stream. 35 00:02:56,220 --> 00:03:00,440 I'm going to do a quick audio check, make sure that things are working because I have 36 00:03:01,010 --> 00:03:04,720 been known to have problems with that in the past, but it looks like we're good. 37 00:03:05,070 --> 00:03:07,220 We're streaming on YouTube and I can hear it. 38 00:03:07,500 --> 00:03:07,740 Awesome. 39 00:03:09,040 --> 00:03:13,540 Okay, well, I am going to just kind of keep an eye on that because I'm running it on my 40 00:03:13,790 --> 00:03:14,200 laptop today. 41 00:03:14,920 --> 00:03:20,660 My home server, ironically, is not as powerful as my laptop. I need to get a better home server, 42 00:03:21,380 --> 00:03:27,540 but priorities, money, all that stuff. So for now, we're going to run on my laptop and I'm going to 43 00:03:27,620 --> 00:03:34,960 show you what we got. Today we're hoping to cover AI that does text chat. So you can simply just 44 00:03:35,140 --> 00:03:40,640 chat with AI back and forth on your local setup. We're going to look at image generation, possibly 45 00:03:40,900 --> 00:03:44,379 video generation. I'll at least show you kind of some of the tools, but I don't know if we'll 46 00:03:44,280 --> 00:03:52,080 actually do any video gen and then we have audio using whisper which is an open source audio llm 47 00:03:52,959 --> 00:04:00,000 sorry open source audio model and then i'll show you how you can do your own coding locally with 48 00:04:00,620 --> 00:04:06,320 with uh coding models we won't do a lot of it on this stream just because there's only so much time 49 00:04:06,700 --> 00:04:11,319 so let's get into it i'm going to start sharing the screen we've got over 100 people in here 50 00:04:11,320 --> 00:04:17,320 Welcome, everybody. Thanks. If you want to spread the word, just hit that like button. I cringe at 51 00:04:17,359 --> 00:04:22,019 saying that, but whatever. The algorithms need it. So hit that like button. Go and subscribe to this 52 00:04:22,120 --> 00:04:28,740 channel. I'm coming to you over the TFTC channel on YouTube. Truth for the commoner. We align very 53 00:04:28,940 --> 00:04:35,620 closely on our views of freedom tech and trying to get more people taking advantage of these 54 00:04:35,740 --> 00:04:40,100 awesome tools that are out there, using them, provide feedback to the developers, suggest ideas, 55 00:04:40,620 --> 00:04:42,220 And then maybe use some of these tools. 56 00:04:42,360 --> 00:04:46,980 Maybe today you learn what tools you can use to write your own programs, write your own software. 57 00:04:48,040 --> 00:04:50,940 Because you're not a software engineer, but you want to build stuff. 58 00:04:51,120 --> 00:04:55,600 So grab these tools, start building stuff, and you too can make your own freedom technology. 59 00:04:56,340 --> 00:05:01,460 So that's what we're going to try and help people figure out how to do in their lives. 60 00:05:03,100 --> 00:05:03,420 Okay. 61 00:05:04,380 --> 00:05:05,180 We're five minutes in. 62 00:05:05,360 --> 00:05:10,360 Let's start sharing the screen and let's get into all of these local models. 63 00:05:10,580 --> 00:05:14,000 so let me just do this 64 00:05:16,780 --> 00:05:18,340 we're going to share the entire screen 65 00:05:19,370 --> 00:05:20,800 because I'm going to be jumping between apps 66 00:05:20,930 --> 00:05:22,720 and so it'll just be too much to try and 67 00:05:25,880 --> 00:05:27,320 try and just do one window right 68 00:05:28,520 --> 00:05:28,960 okay 69 00:05:30,220 --> 00:05:30,660 screen 70 00:05:32,780 --> 00:05:33,520 there we go 71 00:05:35,340 --> 00:05:36,720 okay looks like it's up 72 00:05:36,940 --> 00:05:39,099 again please feel free to share 73 00:05:39,100 --> 00:05:46,040 anything in the chat, any comments, questions you have. And I'll try to watch for them. I'm not, 74 00:05:46,640 --> 00:05:50,720 I don't have ZapStream open. I guess I could quickly try it out real quick here, 75 00:05:50,960 --> 00:05:54,560 but it's in this browser. Let's see. Looking like ZapStream is working today. 76 00:05:55,660 --> 00:05:59,640 I'm not going to be able to watch the chat just because, well, maybe I can. Let's try. 77 00:06:00,360 --> 00:06:07,759 Let's do a new window. I'll drag it off over here and then I'll mute it. Okay. I'm going to try 78 00:06:07,760 --> 00:06:15,920 and watch the Zap stream chat if anybody hops in there. If not, it's all good. It's all good, man. 79 00:06:20,380 --> 00:06:27,300 Okay. I think I got it. All right. Why run your own local AI, right? You got chat GPT. 80 00:06:28,080 --> 00:06:33,320 They just came out with new models. GPT-5. It's all good. It's super powerful. All that stuff. 81 00:06:34,400 --> 00:06:41,340 You've got Claude code and you can run that in cursor. Why do you want to run your own local 82 00:06:41,440 --> 00:06:49,560 stuff? One of the biggest reasons is freedom and privacy. If you are running something on your 83 00:06:49,600 --> 00:06:54,520 computer and you have the internet turned off, you've got the best privacy. Full stop. 84 00:06:56,080 --> 00:07:01,680 That's where you start. Then you start to look at tradeoffs from there. Maybe your computer is not 85 00:07:01,680 --> 00:07:08,420 as powerful as a cloud server. So what do you do? Do you buy a cloud server and put it in your home? 86 00:07:08,860 --> 00:07:14,140 That is an option. It might be expensive. It might require you to do a lot of tinkering and 87 00:07:14,360 --> 00:07:18,260 updating yourself. Maybe you have some power requirements where you have to bring extra, 88 00:07:18,720 --> 00:07:24,300 you know, get an electrician or do some electrical work to get more power to that GPU because the 89 00:07:24,440 --> 00:07:30,819 cloud-based GPUs require much, much beefier hardware or power requirements. And they make 90 00:07:30,640 --> 00:07:35,980 make a lot of noise. Those fans are very loud. I have been there in the data centers with these 91 00:07:36,380 --> 00:07:44,160 NVIDIA chips that are running and they're super loud. So then you look at other trade-offs like, 92 00:07:44,360 --> 00:07:51,640 can I use a cloud AI? And if you're going to use something like Grok or ChatGPT, 93 00:07:52,380 --> 00:07:56,859 you are giving them all your data. Even if you use the incognito mode and say this is a private 94 00:07:56,860 --> 00:08:01,640 chat, what they do is they actually just hide that from the UI after you're done chatting. 95 00:08:01,710 --> 00:08:05,740 Like it disappears from the UI. But if you look at their retention policies for data retention, 96 00:08:06,280 --> 00:08:12,400 they retain those chats for at least 30 days. Most of them do. ChatGPT has recently had a lawsuit 97 00:08:12,570 --> 00:08:17,000 where they're being ordered to retain them for longer. And those chats also make their way into 98 00:08:17,220 --> 00:08:21,979 the models that get trained. And so, they're really they're retained indefinitely because 99 00:08:21,980 --> 00:08:27,620 they become part of that corpus. So we have a middle ground. This will be my quick pitch here 100 00:08:27,920 --> 00:08:34,039 of Maple AI. We provide a chat GPT experience, but you're fully encrypted end-to-end. So your device 101 00:08:34,479 --> 00:08:39,900 uses a private encryption key using our secure servers, the secure enclaves, and then runs the 102 00:08:40,030 --> 00:08:51,960 AI in the cloud. It's also secure enclave with the GPU. These are all open source models. None 103 00:09:24,160 --> 00:09:26,680 then let's say that they are proxying to OpenAI. 104 00:09:27,310 --> 00:09:28,380 Well, what you're depending on then 105 00:09:28,580 --> 00:09:31,840 is that they are just kind of mixing your chat 106 00:09:31,930 --> 00:09:32,840 with everybody else's chats, 107 00:09:33,030 --> 00:09:34,140 and so you get lost in the crowd. 108 00:09:34,770 --> 00:09:37,580 But the moment that you put any personal information 109 00:09:37,910 --> 00:09:40,040 in that chat, you put your name in there, 110 00:09:40,220 --> 00:09:42,060 you put something that is identifiable 111 00:09:42,500 --> 00:09:44,260 for your location or something, 112 00:09:45,550 --> 00:09:47,460 then OpenAI now knows about you. 113 00:09:48,340 --> 00:09:54,300 So you're only safe, you know, in the crowd until you self-identify somehow. 114 00:09:54,500 --> 00:10:01,300 And it's easier than you expect because you just get in the flow of chatting and you out yourself in some way. 115 00:10:01,880 --> 00:10:05,520 So with Maple, you are in your own private vault. 116 00:10:05,780 --> 00:10:09,060 All of your data is in a vault with a secure enclave. 117 00:10:09,560 --> 00:10:11,020 You're chatting with open source models. 118 00:10:11,620 --> 00:10:17,200 We have built the closest thing to installing your own cloud server in your home. 119 00:10:17,700 --> 00:10:23,080 closest thing as possible that we can. But the tradeoff is still that you are trusting our 120 00:10:23,250 --> 00:10:29,480 secure enclaves and it's running in the cloud, but then this is open source. So you can go look at 121 00:10:29,490 --> 00:10:32,640 our GitHub. You can see all the code that's running on the server. You can build the code 122 00:10:32,800 --> 00:10:36,840 yourself. You can get that checksum and compare it against what we are running the secure enclaves, 123 00:10:36,970 --> 00:10:42,220 and you can verify that we are not logging your stuff. Okay. So let's get into it. 124 00:10:42,660 --> 00:10:49,900 Some of my favorite tools. Real quick, somebody, when I posted this, shared this blog post here, 125 00:10:50,400 --> 00:10:53,620 Average Gary did. I mentioned that you can build your own home server. 126 00:10:54,620 --> 00:10:59,140 This looks like a great option. If you really want to go and get a huge beefy thing going, 127 00:10:59,330 --> 00:11:04,020 you can build your own cluster at home with multiple computers, wire them together so they 128 00:11:04,080 --> 00:11:11,860 can run massive open source models. This one's using the framework, uh, framework base. And, 129 00:11:12,280 --> 00:11:21,940 um, I wanted to get down to the cost. Where was it down here? So there's, they spent about $8,000 130 00:11:23,420 --> 00:11:31,000 on that. And then you can get like an M4 pro mini cluster spending $20,000. But this is an 131 00:11:30,960 --> 00:11:34,780 interesting thing if you want to do. For the most part, they're still slower than the cloud ones, 132 00:11:35,200 --> 00:11:39,200 but it's all about trade-offs, right? And this is a very new blog post. It's just from yesterday. 133 00:11:40,680 --> 00:11:44,220 Okay, let's go into some of my favorite tools, and let's look at this question here. 134 00:11:44,740 --> 00:11:48,480 Is there a way to request your chat data from OpenAI if you realize you don't want them having 135 00:11:48,560 --> 00:11:57,740 all of your data? So there are a couple things you can do with OpenAI. OpenAI data delete. 136 00:11:58,980 --> 00:12:12,360 I'm not going to log into my chat GPT right now on the stream, but go look at here and you can see how you can submit to have all of your data deleted, which is kind of the final step, right? 137 00:12:13,180 --> 00:12:20,960 But prior to that, you can actually go into OpenAI and you can look at there's like a memory feature where you can see what it knows about you. 138 00:12:21,600 --> 00:12:25,700 And it'll show you a box and say, here's all the things that I know about this person as you've chatted over the months. 139 00:12:26,300 --> 00:12:30,400 then their AI says, oh, this thing that they said seems really important. That is their 140 00:12:30,940 --> 00:12:36,220 date of birth or their weight or it's their name or the city they live in and starts to remember 141 00:12:36,390 --> 00:12:40,600 these things, right? So, you can go in there and you can actually delete stuff from that if you 142 00:12:40,610 --> 00:12:46,500 want to. That only affects your chats. And that's also only the thing that they show to you. We have 143 00:12:46,510 --> 00:12:49,720 no guarantee if you delete it out of there, if it actually gets deleted out of their system. 144 00:12:50,220 --> 00:12:54,220 It's very likely and very possible that they have kind of like some secondary 145 00:12:55,480 --> 00:12:59,300 view of you that they don't share with you and that they are constantly compiling. 146 00:13:00,580 --> 00:13:07,780 So, really the best way is to go in and go into the data controls and submit to have your data 147 00:13:08,040 --> 00:13:13,180 deleted completely. But that's like a full score stir if you're deleting everything from ChatGPT. 148 00:13:14,040 --> 00:13:17,980 But you can do that and then start over with a new one. I use ChatGPT, but I'm very careful. 149 00:13:18,060 --> 00:13:24,800 I don't chat about anything personal or private. I simply use it for specific business things 150 00:13:25,220 --> 00:13:29,380 that I'm doing. I use Maple for everything else. I also use Maple for business things. 151 00:13:31,240 --> 00:13:36,020 We're doing a fundraise right now. We've been doing a lot of strategy talk. And I've been 152 00:13:36,240 --> 00:13:41,120 using Maple. I've been using DeepSeek, this big one, 671B. I've been using the new OpenAI 153 00:13:41,280 --> 00:13:47,499 open source that came out this week to do it. So it's very powerful. I only use ChatGPT for things 154 00:13:47,500 --> 00:13:52,740 that we simply just don't have in Maple like image generation if I need powerful image gen online. 155 00:13:53,900 --> 00:13:58,560 So hope that helps. Right on. Okay. So let's look at some of the tools. The one that I use the most 156 00:13:58,680 --> 00:14:04,920 for running my local AI is called LM Studio. It's super easy. And it's available on every platform. 157 00:14:05,140 --> 00:14:09,780 You can get it on Mac, Windows, Linux. So I use the Mac version obviously because I'm on a Mac. 158 00:14:10,500 --> 00:14:15,680 But let me show you what that is. So this is LM Studio. Let's see. 159 00:14:20,679 --> 00:14:21,940 How do I get fresh? 160 00:14:24,079 --> 00:14:25,340 I don't know what's going on here. 161 00:14:26,880 --> 00:14:27,160 All right. 162 00:14:27,680 --> 00:14:29,800 Let me get a new one open here. 163 00:14:29,860 --> 00:14:30,740 I don't know what's going on. 164 00:14:31,520 --> 00:14:32,140 Clear all messages. 165 00:14:32,500 --> 00:14:32,780 Sure. 166 00:14:33,300 --> 00:14:33,620 There we go. 167 00:14:33,900 --> 00:14:34,540 Okay, let's start fresh. 168 00:14:36,679 --> 00:14:37,480 So, hello. 169 00:14:41,819 --> 00:14:42,940 I don't have a model loaded. 170 00:14:43,300 --> 00:14:43,400 Okay. 171 00:14:43,760 --> 00:14:45,500 So, I tried typing to it and nothing happened. 172 00:14:45,580 --> 00:14:51,040 So I go into my model loader and what do we got here? Well, let's go look at my models 173 00:14:51,300 --> 00:14:54,880 So these are the models that I have downloaded. I recently downloaded the open AI one 174 00:14:55,080 --> 00:14:58,180 That's what we're gonna look at today. Mostly. It's the 20 B and 175 00:14:58,760 --> 00:15:02,400 Then if you look over at maple here, we have open AI 176 00:15:03,320 --> 00:15:05,120 120 B what's the difference there? 177 00:15:05,500 --> 00:15:06,600 so this is 178 00:15:06,800 --> 00:15:12,900 effectively the number of parameters the the size if you will but the amount of stuff you can do and the amount of stuff the 179 00:15:12,900 --> 00:15:19,700 of brain power that it has to run your your query dive into this more i'm doing like really high 180 00:15:19,780 --> 00:15:24,820 level on that but dive into that more if you want a more technical understanding and i'm happy to 181 00:15:24,880 --> 00:15:31,220 chat online about it if you want to want to go deeper but um so the smaller one basically the 182 00:15:31,300 --> 00:15:37,320 smaller this is 20 billion parameters this means that it will run on a smaller device like your 183 00:15:37,380 --> 00:15:42,500 computer because the more parameters you have the more ram you need the more memory you need on your 184 00:15:42,340 --> 00:15:47,600 computer right so you can see this is a 12 gigabyte model and it has to load this entire 185 00:15:47,870 --> 00:15:56,200 thing into ram and i'm gonna do something a little dicey here let's just open up let's see 186 00:15:56,260 --> 00:16:01,980 so here is my i move the processes off the screen but here's my my memory usage on my computer i've 187 00:16:01,980 --> 00:16:10,720 got 36 gigs of ram on this laptop currently using 30 gigabytes um and then i've got some swap swap 188 00:16:10,720 --> 00:16:16,180 is just the hard drive that you have. So if it runs out of RAM, then it uses the hard drive as 189 00:16:16,320 --> 00:16:21,900 RAM. And on newer computers now that we have these solid state drives, it's really close in speed to 190 00:16:22,100 --> 00:16:26,160 RAM itself. So it's much better than the old days when you had these platters that were spinning and 191 00:16:26,300 --> 00:16:31,940 you had to page out to the hard drive. So it's a little bit better. So you might think I'm not 192 00:16:31,980 --> 00:16:36,520 going to be able to run this, but what it'll do is Mac OS will boot some other things out of memory 193 00:16:36,520 --> 00:16:40,340 that it needs and then it'll start paging out to my SSD and give me more RAM that way. 194 00:16:42,579 --> 00:16:46,980 So again, I'm running the smaller one locally. I don't have enough processing power on this 195 00:16:47,020 --> 00:16:51,700 laptop to run the 120B and so that's why it's nice to have a cloud option available as well. 196 00:16:54,600 --> 00:17:01,720 And how do I get these? Well, let me show you. Super easy. LLM Studio just lets you download 197 00:17:01,720 --> 00:17:08,260 stuff so i go to search and you're like okay i want quinn coder a powerful 30 billion moe coding 198 00:17:08,579 --> 00:17:14,280 model from alibaba quinn joining us larger 480b counterpart right so this one would not run on 199 00:17:14,280 --> 00:17:18,819 your laptop unless you had the most amazing thing in the world you probably need a home server for 200 00:17:18,860 --> 00:17:24,980 that or or group a bunch of laptops together and then it's got these different formats you can 201 00:17:24,980 --> 00:17:30,120 look these up mlx is the one that i'm running because that is what uh is the uh the metal 202 00:17:30,680 --> 00:17:37,300 Apple's metal stuff, right? Here we go. MLX, a new AI ML framework for Apple silicon by Apple. 203 00:17:37,830 --> 00:17:43,780 So it supports that, which is great. I can just hit download and it'll start downloading it. 204 00:17:45,940 --> 00:17:51,800 I got enough space. Let's see. I've got 79 gigs available. Let's just hit download 205 00:17:52,020 --> 00:17:57,080 while we're going through other stuff. Okay. So this will show you what it does. So it just 206 00:17:57,080 --> 00:18:01,600 gives you a really nice easy thing. Where do people typically get models from? Let's look at 207 00:18:01,600 --> 00:18:06,480 that. So we have this website. The most popular place is called Hugging Face. Hugging Face is 208 00:18:06,580 --> 00:18:12,200 this community where people will post open source models and it's kind of like a GitHub for models 209 00:18:12,310 --> 00:18:16,580 if you will. So obviously when OpenAI dropped their two open source models they very quickly 210 00:18:16,750 --> 00:18:17,560 were showing up on 211 00:18:17,560 --> 00:18:18,720 a Hugging 212 00:18:18,720 --> 00:18:21,600 Face and you can see how many downloads they've had. The hearts that 213 00:18:21,600 --> 00:18:27,320 they got. Here is Quinn image, which we're going to look at today as well. So these are the biggest 214 00:18:27,440 --> 00:18:33,020 ones, most popular, but you can browse millions of models on here. Hugging face can be super 215 00:18:33,280 --> 00:18:39,160 overwhelming for a newcomer, right? You come in and you're like, okay, I don't know what all these 216 00:18:39,220 --> 00:18:43,560 things are. And maybe there's like a whole bunch of different versions because people will take 217 00:18:43,720 --> 00:18:48,760 this version of open AI and then they will do some transfer learning on it. They will train it in 218 00:18:48,720 --> 00:18:52,560 their own way. They'll do something and they'll repost it. So now you don't know which one to get. 219 00:18:54,300 --> 00:18:58,900 So you need to kind of get versed on Hugging Face if you really want to come in here. It's 220 00:18:59,460 --> 00:19:04,120 great. It's a great place. I recommend people check it out more. But if you're a more casual 221 00:19:04,400 --> 00:19:08,400 AI user and you just want to use this stuff, that's where these other tools come in. And that's 222 00:19:08,400 --> 00:19:15,019 where LM Studio is great for text-based chat. You just come in and they kind of do some of the 223 00:19:15,000 --> 00:19:20,540 curation for you, right? So you can come in here and see here's all the different ones that they 224 00:19:20,740 --> 00:19:26,620 have. All right. So I'd recommend starting here. If you want to get more technical, go get more 225 00:19:26,680 --> 00:19:35,480 technical on Hugging Face. The next one is Comfy, which is for ImageGen. And this is a great place 226 00:19:35,640 --> 00:19:40,940 to run things like Quinn Image or other image generation models. We're going to look at that 227 00:19:40,940 --> 00:19:46,760 here in a bit. And then you have Whisper, which is an audio model, also from OpenAI. There are 228 00:19:46,830 --> 00:19:52,640 other ones, but this one has been pretty easy to use. And the way that I like to use Whisper for 229 00:19:52,760 --> 00:20:00,120 doing audio on my computer is I use a tool called Mac Whisper. I don't go deep into this very often 230 00:20:00,170 --> 00:20:07,659 with people because Mac Whisper is closed source. So it's not like full Freedom Tech because it is 231 00:20:07,600 --> 00:20:15,680 closed source. It'll also let you use the actual API to talk directly to OpenAI or Anthropic or 232 00:20:15,760 --> 00:20:20,600 others if you want to. I just run it locally. I don't talk back to the cloud servers. 233 00:20:21,740 --> 00:20:27,300 But that's how I do dictation and other things is I use Mac Whisper. But it uses the open source 234 00:20:27,480 --> 00:20:31,060 model. So you can go look on GitHub at the Whisper model. You can see how it's built, 235 00:20:31,140 --> 00:20:38,300 that kind of stuff. And then I thought Comfey did video as well because I did, you know, 236 00:20:38,380 --> 00:20:42,740 put in the thumbnail here that we're going to look at video. So, yeah. Yeah. So, you can do 237 00:20:42,780 --> 00:20:48,580 video on Comfey as well. And then obviously the last one is code. We're downloading 238 00:20:49,220 --> 00:20:52,280 quen coder right now but i wanted to show you 239 00:20:55,179 --> 00:20:58,340 i lost it um quen coder 240 00:21:00,940 --> 00:21:04,160 was that here oh yes i already have it open okay 241 00:21:05,590 --> 00:21:09,480 so here's quen code you can go on github and look at it they've got lots of stars 242 00:21:11,160 --> 00:21:18,020 but you can download this and run your own ai coding tool right on your laptop if you want to 243 00:21:18,020 --> 00:21:26,120 using Quinn. Quinn code 3 is pretty powerful. A lot of people are using it. For building Maple, 244 00:21:26,270 --> 00:21:30,460 we leverage Claude code a lot. We're trying out the new chat GPT stuff they came out with 245 00:21:30,600 --> 00:21:37,280 yesterday. We use cursor with Claude a lot as well as Claude code. Those are super powerful. 246 00:21:38,190 --> 00:21:42,320 We're also looking at Quinn. It would be great to do some more stuff locally for building it. 247 00:21:42,980 --> 00:21:49,040 but we build an open source tool. So, in some ways, it's already going to be slurped up by 248 00:21:51,720 --> 00:21:59,760 these LLMs in the future. So, we make that trade off. All right. Let's go back and look at LM 249 00:21:59,900 --> 00:22:03,100 Studio because I would love to actually load a model and show you how it works. 250 00:22:04,100 --> 00:22:11,640 Okay. So, we're here. Let's go back to our chat. Let's just drag that down there. Okay. 251 00:22:11,700 --> 00:22:17,980 Okay. Select a model up here. Here's where I come to select my model. I'm going to do the open 252 00:22:18,240 --> 00:22:25,420 source of chat GPT load model. Okay. So it's loading. It's pretty quick. It still takes a 253 00:22:25,500 --> 00:22:29,140 while though. If we were to look at my memory footprint again, let's see what happens here. 254 00:22:29,750 --> 00:22:35,320 That says load in the model. So it did free up some memory and then it's now loading and we see 255 00:22:35,320 --> 00:22:41,600 we're hitting we're starting to hit memory pressure as we put this model in okay so now 256 00:22:41,600 --> 00:22:46,220 we're starting to page out swap uses over three gigabytes it was just barely above one earlier 257 00:22:47,820 --> 00:22:53,920 our real memory use is 32 oh sorry physical yeah 32 excuse me 258 00:22:56,120 --> 00:22:58,640 okay the model is loaded now so let's say hello 259 00:23:01,220 --> 00:23:05,680 okay so it's thinking what does it mean by thinking okay hello how can i assist you today 260 00:23:06,420 --> 00:23:11,940 so this is reasoning effort lm studio lets you look at does the model reason what does reasoning 261 00:23:12,040 --> 00:23:17,560 mean well reasoning is where the model like is introspective and looks at your thing and says 262 00:23:17,760 --> 00:23:23,220 okay here's what they said let me build a prompt for myself based off what they said and then i 263 00:23:23,200 --> 00:23:28,980 will run the query. Whereas the earlier versions of ChatGPT simply just ran your query against the 264 00:23:29,040 --> 00:23:34,980 LLM. So if you do low, you're going to get almost no thinking at all. How can I assist you today? 265 00:23:35,580 --> 00:23:41,220 What things are you good at? So I turned off. I made reasoning really low. And so it doesn't 266 00:23:41,300 --> 00:23:46,600 think for any seconds now. It just spits things out to me. So this model is good at writing and 267 00:23:46,700 --> 00:23:51,700 editing, drafting emails, that kind of stuff, reports. It's good at research and summaries, 268 00:23:52,580 --> 00:23:53,720 coding help, learning tutorials. 269 00:23:53,910 --> 00:23:55,260 So here's all the stuff that it's good at. 270 00:23:56,210 --> 00:23:57,700 And it was pretty fast, right? 271 00:23:57,810 --> 00:24:00,300 It gave us 32 tokens per second. 272 00:24:01,020 --> 00:24:03,400 I had this one was 357 tokens total. 273 00:24:03,940 --> 00:24:06,360 And the time to first token was 0.62 seconds. 274 00:24:06,780 --> 00:24:09,000 These are all metrics that people track a lot 275 00:24:09,160 --> 00:24:10,740 when they're running LLMs. 276 00:24:11,120 --> 00:24:14,560 When you run this in the cloud, this is all so much faster. 277 00:24:14,890 --> 00:24:18,340 But this one's pretty quick for being an open source model 278 00:24:18,340 --> 00:24:19,140 that you're running locally. 279 00:24:20,140 --> 00:24:21,840 Now, if I wanna turn on reasoning more, 280 00:24:24,880 --> 00:24:28,860 build me a business plan for a lemonade stand. 281 00:24:30,300 --> 00:24:32,260 Whoops, I misspelled that, but it's okay if you misspell 282 00:24:32,420 --> 00:24:33,800 because it knew what I was saying anyways. 283 00:24:34,620 --> 00:24:36,700 See, it already, like you can see what it's saying. 284 00:24:37,080 --> 00:24:39,520 Oh, come on, I want to view this. 285 00:24:40,090 --> 00:24:40,420 There we go. 286 00:24:41,220 --> 00:24:42,300 Okay, the user says, 287 00:24:42,620 --> 00:24:44,280 build me a business plan for lemonade stand. 288 00:24:44,350 --> 00:24:45,260 So it corrected my spelling. 289 00:24:46,140 --> 00:24:47,780 And then, you know, it says, 290 00:24:47,870 --> 00:24:49,460 they likely want a comprehensive business plan. 291 00:24:49,980 --> 00:24:51,400 So you can see like what it's doing, 292 00:24:51,580 --> 00:24:52,300 what it's thinking through, 293 00:24:52,920 --> 00:24:54,400 and then it's building its own prompt 294 00:24:55,000 --> 00:24:56,800 based off of what I asked it. 295 00:24:57,960 --> 00:24:58,660 So it's still thinking. 296 00:25:01,000 --> 00:25:02,520 Okay, now it's giving me my business plan, 297 00:25:02,700 --> 00:25:03,380 spitting it out here. 298 00:25:04,760 --> 00:25:07,840 If I was doing this in Maple, it'd be much faster. 299 00:25:07,890 --> 00:25:09,760 In fact, we could do a side-by-side comparison 300 00:25:09,880 --> 00:25:12,440 if we want to see how fast the bigger model is 301 00:25:12,470 --> 00:25:13,240 to the smaller model. 302 00:25:15,870 --> 00:25:16,580 My download is complete. 303 00:25:17,070 --> 00:25:18,560 All right, so I've got QuenCoder 3, 304 00:25:49,660 --> 00:25:56,160 I currently have reasoning turned off for this version. 305 00:25:56,800 --> 00:25:59,360 So maybe I'll do this without any reasoning. 306 00:25:59,620 --> 00:26:00,100 Let's do that. 307 00:26:01,020 --> 00:26:05,380 I wanna do like a true comparison. 308 00:26:07,180 --> 00:26:07,660 Clear. 309 00:26:08,540 --> 00:26:09,260 Okay, let's start fresh. 310 00:26:11,980 --> 00:26:13,340 Put the reasoning left for it down to low. 311 00:26:14,660 --> 00:26:16,000 Correct my spelling mistake. 312 00:26:16,760 --> 00:26:19,080 Okay, so we're gonna do a side-by-side speed test 313 00:26:19,080 --> 00:26:24,600 between GPT-OSS, which stands for open source software, 314 00:26:25,580 --> 00:26:29,260 120B versus the 20B, which I'm running locally on my laptop. 315 00:26:30,620 --> 00:26:34,140 This one's running on Maple on powerful Nvidia GPUs. 316 00:26:35,539 --> 00:26:35,940 Okay. 317 00:26:39,179 --> 00:26:40,700 There we go, all right, it's ready to go. 318 00:26:41,500 --> 00:26:44,420 So I'm just gonna hit return and go. 319 00:26:45,820 --> 00:26:45,940 Okay. 320 00:26:48,860 --> 00:26:53,800 So they're both spitting out pretty similar as far as content goes. 321 00:26:55,420 --> 00:26:57,480 Obviously Maple is ahead. 322 00:26:59,000 --> 00:27:00,320 I didn't do this before the stream. 323 00:27:00,640 --> 00:27:05,360 I was kind of worried that it wasn't going to be faster, but in theory it should be faster. 324 00:27:06,780 --> 00:27:09,980 So this is on the operations plan still on number five. 325 00:27:10,160 --> 00:27:12,360 We're on number seven over here on the left. 326 00:27:15,560 --> 00:27:16,040 Okay. 327 00:27:16,320 --> 00:27:24,620 done. This one's still running. Now in Maple, we don't show you your like tokens per second or 328 00:27:24,680 --> 00:27:30,980 anything. But it feels very much, it's much faster. So I can be getting more done here. 329 00:27:31,060 --> 00:27:36,660 I can already be chatting with it more. How about for a trading card stand? 330 00:27:39,880 --> 00:27:45,100 Okay, this is done now. So we took, we got around 30 tokens per second, which is pretty good. 331 00:27:46,419 --> 00:27:53,180 We didn't do any reasoning. Let's see. This was 1400 tokens and time to first token was 332 00:27:53,680 --> 00:28:00,760 half a second, roughly. Pretty good. Okay. So, that's local AI. Again, you can change the model 333 00:28:00,960 --> 00:28:05,460 you're running. Let's switch to CoinCoder. Let's just do that really quick while we're here. 334 00:28:06,120 --> 00:28:12,380 Context length. Okay. So, what's context length? This is how many, basically how much text you 335 00:28:12,380 --> 00:28:17,780 want to pass in. For lack of a better way to explain that. Just the amount of text you want 336 00:28:17,780 --> 00:28:22,500 to pass in. Setting a high value for context link can significantly impact memory usage. 337 00:28:23,250 --> 00:28:27,060 And that's because whatever you pass into it, it has to load it all into RAM 338 00:28:28,280 --> 00:28:34,760 for the LLM to process it. Let's hit load. Okay. So, for example, let's scroll back on 339 00:28:34,820 --> 00:28:39,880 Maple while this is loading. All of this stuff here is now context. Like, everything you see, 340 00:28:41,360 --> 00:28:45,200 even the first question I asked it, and if I continue to ask it follow-up questions, 341 00:28:45,850 --> 00:28:50,220 everything you see on here becomes context within the LLM. So, the longer that I chat, 342 00:28:50,590 --> 00:28:57,480 I need some water. The longer that I chat, the more context it's getting, which is helpful, 343 00:28:57,950 --> 00:29:01,580 but also the more expensive it gets, the more computing power you need, the more memory you need. 344 00:29:02,280 --> 00:29:10,000 It just, it makes everything bigger. Okay. Is this loaded? I think it's loaded. Okay. Let's 345 00:29:10,000 --> 00:29:13,960 Oh, there's the button. When I'm on screen, I kind of forget stuff sometimes. 346 00:29:15,220 --> 00:29:15,560 All 347 00:29:15,560 --> 00:29:17,760 right. We're up to over 500 people watching. Welcome, everybody. 348 00:29:18,720 --> 00:29:22,340 It's a wonderful Friday. This is Freedom Tech Weekend. If you're just joining us, I am not 349 00:29:22,480 --> 00:29:30,360 Marty. I'm not Odell. I am Marks. I run, I'm a co-founder of Maple AI, secure encrypted AI in 350 00:29:30,360 --> 00:29:35,280 the cloud. You can go to try maple.ai. Today, we're talking about how to run your own local AI. 351 00:29:35,600 --> 00:29:38,280 So we're running things locally with LM Studio. 352 00:29:38,880 --> 00:29:41,740 We've got Comfy UI for doing image generation. 353 00:29:42,050 --> 00:29:43,740 We have Mac Whisper for doing audio. 354 00:29:44,400 --> 00:29:45,880 Comfy UI also does video. 355 00:29:46,110 --> 00:29:48,740 And now we are about to try out coding. 356 00:29:49,540 --> 00:29:51,320 So let's do something simple. 357 00:29:52,140 --> 00:29:55,960 Let's say my favorite, 358 00:29:56,310 --> 00:29:58,920 because I often have to download YouTube videos for clips. 359 00:30:00,540 --> 00:30:01,260 My own stuff. 360 00:30:01,490 --> 00:30:02,880 I'm not doing anything bad. 361 00:30:03,560 --> 00:30:08,660 But let's say write me, I don't know, I don't want to, what do I want to say here? 362 00:30:09,760 --> 00:30:14,480 How about we want to convert, we want to convert audio. 363 00:30:14,940 --> 00:30:20,200 So let's say I recorded some audio in M4A, but I want to convert it to MP3. 364 00:30:20,760 --> 00:30:32,440 Okay, write me a JavaScript that will convert M4A audio file to MP3 audio file. 365 00:30:32,860 --> 00:30:35,360 Let's see what it does with this. I don't actually know what it'll do here. 366 00:30:37,160 --> 00:30:41,200 Okay, so it's loading CoinCoder. Here's a JavaScript solution. Okay, so it's going to 367 00:30:41,200 --> 00:30:48,520 use FFmpeg, which is understandable. And then here's my JavaScript file. 368 00:30:49,970 --> 00:30:54,080 What is it using? How am I? It should give me... Okay, so here it's telling me how to install 369 00:30:54,660 --> 00:30:58,740 FFmpeg. This is awesome. It's like your own little mini GitHub. It gives you installation 370 00:30:58,740 --> 00:31:03,560 steps and everything as you work through the different systems you might be on. And then 371 00:31:03,680 --> 00:31:06,660 it's going to have you use Node. So I'll have to install Node on my computer if I don't have it. 372 00:31:06,740 --> 00:31:14,440 I already have it. It tells you how to run it. So that's pretty cool. I'm not going to dive deep 373 00:31:14,520 --> 00:31:19,540 into this right now, but you can just grab LM Studio, download CoinCoder 3 through the UI, 374 00:31:19,760 --> 00:31:25,700 and start writing your own stuff. Super basic way to start. Okay. So whatever job you're working, 375 00:31:27,300 --> 00:31:33,140 If you're working full time and you're like, man, I could really use some little tool to help me do 376 00:31:33,200 --> 00:31:37,640 this thing. Maybe you get a lot of data from somewhere at work and you're like are always 377 00:31:37,840 --> 00:31:43,500 copying and pasting and doing manual conversion or you're doing manual analysis on it, throw it in 378 00:31:43,520 --> 00:31:47,580 here. Download Elum Studio. Throw it in here. You can attach documents, right? I can come in here 379 00:31:48,140 --> 00:31:55,180 and I can attach stuff. Toss it in there. Have it write a script to a JavaScript or some program. 380 00:31:55,220 --> 00:32:01,160 you can probably have it write an app. Can this write iPhone apps? Make it a macOS native 381 00:32:01,400 --> 00:32:08,380 application instead. I don't know what this will do. Okay, so it's making an electron app. 382 00:32:11,660 --> 00:32:15,740 That's fine. So it's not going to use Swift. I probably could do Swift, but it's not going to 383 00:32:15,840 --> 00:32:20,960 here, probably because I started with JavaScript, and so it's just taking that. But you could build 384 00:32:21,000 --> 00:32:24,340 your own little app that runs on your computer, and then you like drag and drop files onto it and 385 00:32:24,260 --> 00:32:30,320 stuff. Okay. But it's spitting it out. Now, if you're going to be coding full time for your job, 386 00:32:30,640 --> 00:32:34,760 then you would want to get like an actual developer environment, whether that's a command 387 00:32:34,820 --> 00:32:41,060 line interface or an IDE integrated development environment. But this is a great way to get 388 00:32:41,240 --> 00:32:44,960 started for people who just want like one-off things that they're doing to help make the job 389 00:32:45,100 --> 00:32:53,020 better. So you can really kind of show up and show off at your job by using a tool like this 390 00:32:53,020 --> 00:32:59,420 to build little conversion scripts for yourself or utilities for yourself that are highly specific 391 00:33:00,160 --> 00:33:05,300 to your job role. Like you're not going to be able to go online and find some tool that 392 00:33:05,600 --> 00:33:09,820 understands exactly what you do day in and day out and the kind of data you're dealing with. 393 00:33:10,060 --> 00:33:13,720 You're always going to be trying to figure out how do I take this tool and shoehorn it into my work, 394 00:33:14,120 --> 00:33:18,380 right? Well, with this, you can build the tool specifically to your requirements. 395 00:33:19,220 --> 00:33:22,400 You can even have, let's open up a new chat here, 396 00:33:23,040 --> 00:33:24,100 something we do all the time. 397 00:33:26,860 --> 00:33:32,240 Build me a technical spec specification 398 00:33:33,750 --> 00:33:39,320 that I can hand to a coding LLM for this feature. 399 00:33:40,580 --> 00:33:45,320 Okay, so we're gonna say convert audio 400 00:33:47,900 --> 00:33:53,100 from any format into mp3. 401 00:33:57,019 --> 00:34:00,840 Accept audio files that I give you. 402 00:34:04,440 --> 00:34:06,860 Handle multiple files at once. 403 00:34:08,770 --> 00:34:12,200 And last one, we will say, well, let's just do that. 404 00:34:12,800 --> 00:34:12,899 Okay. 405 00:34:13,840 --> 00:34:17,879 So then it builds you a specification 406 00:34:17,879 --> 00:34:22,300 about all these edge cases that I might not have considered. Like, hey, we need to have all these 407 00:34:22,399 --> 00:34:27,320 different bit rates. And it's probably going to go into the different formats. Yeah, here's all 408 00:34:27,320 --> 00:34:31,399 the different formats I might need to worry about. I can read through this and say, okay, 409 00:34:32,720 --> 00:34:38,919 don't do that thing, but yes, do this thing. And then I can add more. I can refine this, right? 410 00:34:39,480 --> 00:34:45,620 And then when I'm done, it's like, all right, I got this whole thing. Now I can come in here, 411 00:34:46,980 --> 00:34:54,460 build a new version of this JavaScript file using the following spec. 412 00:34:56,720 --> 00:35:00,840 I should probably start a new prompt instead of building on the old one, but that's all right. 413 00:35:01,900 --> 00:35:07,160 I might want to start clean and not have the context from before. All right, so it's processing 414 00:35:07,240 --> 00:35:11,100 it. It's taking a little while. Okay, it's having to read this giant prompt I just gave it. 415 00:35:13,060 --> 00:35:14,420 And now it's implemented for me. 416 00:35:15,170 --> 00:35:29,420 Okay, so what we're doing here is we're showing you how to be a product manager, a project manager, like a technical program manager, whatever these PM roles are, right, that you maybe have on your team. 417 00:35:30,010 --> 00:35:38,380 And then we're also showing you how to be your own mini software engineer, a data analyst, like whatever role you might have at your desk job. 418 00:35:39,060 --> 00:35:44,140 If you don't have a desk job, if you are working with your hands, like there are other ways you can use this too to get your job done. 419 00:35:44,980 --> 00:35:46,360 So many things you can do here. 420 00:35:47,480 --> 00:36:00,140 Back in the day, I worked with some guys who would go in and repair homes who had had flood damage or some other problem where insurance was going to give them money to do repairs on their house. 421 00:36:00,440 --> 00:36:04,820 They would come in and assess it and they would have to like bid out really quickly. 422 00:36:05,600 --> 00:36:06,840 And there were multiple people, right? 423 00:36:06,960 --> 00:36:12,840 The homeowner had some leak in their home, a pipe burst, and their floors were ruined. 424 00:36:13,080 --> 00:36:14,980 Maybe a lot of their walls were damaged. 425 00:36:15,660 --> 00:36:17,400 They're calling a bunch of people, bringing them in. 426 00:36:17,540 --> 00:36:26,600 These people all assess the situation, and they need to get back to them as quick as they can with a bid that they can honor, that they're not going to lose money on. 427 00:36:27,610 --> 00:36:32,640 So they want to do the highest price, but also the lowest price, as close as they can get to being efficient. 428 00:36:34,200 --> 00:36:39,860 Well, I helped someone build a little software tool that they could run on their iPad and they could show up in the house. 429 00:36:40,060 --> 00:36:42,580 This was a long time ago before other people had done this. 430 00:36:42,940 --> 00:36:47,080 And they would just go on their iPad and they would tap this, that, here's the damage, here's the size, all that stuff. 431 00:36:47,420 --> 00:36:49,000 And it would give them a quote right there on the spot. 432 00:36:50,060 --> 00:36:51,520 And that was really beneficial to them. 433 00:36:51,560 --> 00:36:53,660 It gave them a huge edge up in their business. 434 00:36:54,180 --> 00:36:58,780 And so they could bring their contractors in and work on jobs, get a lot more jobs. 435 00:37:00,220 --> 00:37:01,960 They could use a tool like this now. 436 00:37:02,120 --> 00:37:03,380 They wouldn't need me necessarily. 437 00:37:03,780 --> 00:37:06,240 They could build their own specification, right? 438 00:37:06,390 --> 00:37:08,680 They could talk to, you and I talked to this 439 00:37:08,790 --> 00:37:09,720 for just a few seconds. 440 00:37:10,420 --> 00:37:13,120 You could spend an entire week, two weeks if you want to, 441 00:37:13,779 --> 00:37:15,940 really homing and refining some of these prompts. 442 00:37:16,640 --> 00:37:18,560 And then you could go and dump it in here 443 00:37:18,610 --> 00:37:20,940 and have it build the app, go back and forth, 444 00:37:21,120 --> 00:37:23,540 iterate on it a ton and build this really kick-ass app 445 00:37:23,610 --> 00:37:24,200 that you can use. 446 00:37:25,080 --> 00:37:26,180 So just trying to give ideas 447 00:37:26,570 --> 00:37:27,980 of how you can do these things on your own. 448 00:37:29,200 --> 00:37:31,420 Okay, but again, it's kind of slow, 449 00:37:33,020 --> 00:37:37,140 Depending on your hardware, if you want to buy your own big beefy server and run it at home, you can. 450 00:37:38,740 --> 00:37:40,960 Or you can use cloud tools as well. 451 00:37:42,300 --> 00:37:43,980 Let's move on to ImageGen. 452 00:37:44,380 --> 00:37:49,460 This is one I have not done very much, and so I'm actually really interested and excited to try it out. 453 00:37:49,660 --> 00:37:52,040 Before we got on the stream, let's see. 454 00:37:52,160 --> 00:37:57,660 What would you use to go from a query to a tabulated data graphic chart of the data model returned? 455 00:37:59,420 --> 00:38:00,380 So let's see. 456 00:38:01,800 --> 00:38:02,320 You know, it's interesting. 457 00:38:02,640 --> 00:38:09,380 So one thing you can do is you're going from a query to tabulated data to graphic chart. 458 00:38:10,100 --> 00:38:16,880 So if you're using spreadsheets, for example, then you can actually have the AI print out 459 00:38:17,400 --> 00:38:19,380 CSV, comma separated values. 460 00:38:20,470 --> 00:38:21,460 So let's do a new one. 461 00:38:21,510 --> 00:38:22,320 Let me show you what I mean. 462 00:38:24,000 --> 00:38:28,340 Actually in Maple, I have an example of this. 463 00:38:28,880 --> 00:38:30,140 So let's open up my history. 464 00:38:30,550 --> 00:38:31,680 So Austin weather data. 465 00:38:31,820 --> 00:38:33,960 I was doing an example one time where I downloaded 466 00:38:36,520 --> 00:38:38,900 the weather in just a text file 467 00:38:39,000 --> 00:38:40,140 and then I uploaded it to Maple 468 00:38:40,280 --> 00:38:42,160 'cause Maple can do documents images. 469 00:38:42,740 --> 00:38:44,080 Our document stuff is paused right now. 470 00:38:44,220 --> 00:38:45,960 We had some issues with the reliability, 471 00:38:46,260 --> 00:38:47,380 but we're bringing it back up soon. 472 00:38:48,340 --> 00:38:50,260 But so I had this document that I put in here 473 00:38:51,260 --> 00:38:53,600 and then I asked it, like, give me some trends. 474 00:38:54,240 --> 00:38:55,700 I could just come in here and say, 475 00:38:56,480 --> 00:38:59,779 give me a CSV output 476 00:39:01,040 --> 00:39:05,120 of the data that I can put into a spreadsheet. 477 00:39:13,740 --> 00:39:15,060 And so now it's thinking 478 00:39:15,210 --> 00:39:16,300 and it's gonna spit that out for me. 479 00:39:17,440 --> 00:39:20,300 Jeff G, I'm a coder, but love AI, changed my life. 480 00:39:20,700 --> 00:39:20,900 Totally. 481 00:39:21,720 --> 00:39:24,220 I've been a software engineer for what, 20 years now? 482 00:39:25,100 --> 00:39:26,860 On and off, I've done other roles, 483 00:39:26,990 --> 00:39:29,380 but I write software as well. 484 00:39:29,720 --> 00:39:31,900 And AI has totally changed it because it used to be 485 00:39:32,020 --> 00:39:33,740 that you were kind of stumbling through, 486 00:39:33,980 --> 00:39:34,820 figuring things out. 487 00:39:34,940 --> 00:39:35,560 You knew the concepts, 488 00:39:35,990 --> 00:39:37,540 but every language was slightly different. 489 00:39:37,630 --> 00:39:38,440 You had to learn that syntax. 490 00:39:38,690 --> 00:39:39,460 You had to learn an API. 491 00:39:39,830 --> 00:39:43,580 You had to understand all the ins and outs and intricacies. 492 00:39:43,900 --> 00:39:46,460 Now you can just say, hey, like build me this thing. 493 00:39:47,140 --> 00:39:48,760 And then you can look at the code. 494 00:39:48,930 --> 00:39:50,180 You can tinker and tweak it. 495 00:39:50,640 --> 00:39:51,240 It's awesome. 496 00:39:51,540 --> 00:39:52,300 It's really phenomenal. 497 00:39:52,830 --> 00:39:53,480 All right, so here we go. 498 00:39:53,560 --> 00:39:56,700 So now I just, I could copy this and save it. 499 00:39:57,420 --> 00:40:02,120 you know, I open up my text editor. I don't want to do that right now because I don't know what's 500 00:40:02,150 --> 00:40:07,420 in there, but I could open my text editor, save this as a.csv file, and then you can open this 501 00:40:07,510 --> 00:40:13,500 up in any spreadsheet program, Excel, whatever. And then from there, you can make charts, that 502 00:40:13,510 --> 00:40:19,700 kind of stuff from that data. So I hope that's helpful with what you're looking for. Okay, 503 00:40:20,270 --> 00:40:25,580 image generation. Let's try it out. So I just clicked on the first workflow that they had, 504 00:40:26,140 --> 00:40:28,620 and they already have this thing in here. 505 00:40:29,260 --> 00:40:30,240 This is a little more technical 506 00:40:31,040 --> 00:40:32,340 than what we were seeing with LM Studio. 507 00:40:33,100 --> 00:40:35,400 It's kind of showing you the model that you're using, 508 00:40:36,120 --> 00:40:37,380 the prompts that you have. 509 00:40:38,460 --> 00:40:40,000 You can tweak all sorts of stuff. 510 00:40:40,480 --> 00:40:41,260 We're not going to do all this, 511 00:40:41,420 --> 00:40:42,280 but it gets very technical. 512 00:40:43,060 --> 00:40:45,820 So beautiful scenery, nature, glass, bottle, landscape. 513 00:40:46,480 --> 00:40:47,920 Let's just hit go and see what happens. 514 00:40:48,800 --> 00:40:49,260 Where's run? 515 00:40:55,020 --> 00:40:55,980 Run is somewhere. 516 00:40:56,060 --> 00:41:02,720 oh it's underneath there we go okay y'all can see it but i couldn't because 517 00:41:03,670 --> 00:41:04,900 my streaming software was covering it 518 00:41:06,900 --> 00:41:09,500 okay so it's running did it run 519 00:41:11,860 --> 00:41:18,640 oh okay there we go so it ran it um let's do another prompt so i got that little glass thing 520 00:41:19,240 --> 00:41:21,079 let's do another one um 521 00:41:21,080 --> 00:41:40,060 I wanted to watch it go. Browse templates. Image generation. New fresh one. Let's do a guy sitting there at home chatting with a robot. 522 00:41:42,760 --> 00:41:46,960 okay so it's showing you as it goes through it's doing like the sample stuff 523 00:41:50,340 --> 00:41:55,660 so we're at 57 and 15 25 i don't know what the two different percentages are but 524 00:41:56,700 --> 00:42:03,100 it's doing stuff here let's go look at so i've got a lot of memory going on i should close lm 525 00:42:03,240 --> 00:42:08,379 studio which i'm going to do actually let's free up some ram all right so we got this image here 526 00:42:08,799 --> 00:42:17,760 let's zoom in I mean it's decent right it's not as good as chat GPT I could 527 00:42:17,820 --> 00:42:21,220 tweak things right because chat GPT has done a lot of tweaking on these so I can 528 00:42:21,320 --> 00:42:24,340 up the number of steps I wanted to take so how many times is it gonna go over 529 00:42:27,079 --> 00:42:34,919 different things you can do there I did download the new Quinn Quinn image how 530 00:42:34,920 --> 00:42:35,760 How do I load that? 531 00:42:36,160 --> 00:42:36,800 Let's see really quick. 532 00:42:37,800 --> 00:42:38,860 There's gotta be a way to load that. 533 00:42:41,160 --> 00:42:41,800 Load checkpoint. 534 00:42:44,020 --> 00:42:44,880 Can I just drag it? 535 00:42:48,599 --> 00:42:48,920 No. 536 00:42:54,180 --> 00:42:56,440 Well, if we can't figure it out on screen, that's all right. 537 00:42:56,980 --> 00:42:57,780 I wanted to though. 538 00:42:59,859 --> 00:43:00,600 Can we do video? 539 00:43:01,420 --> 00:43:01,880 Let's try video. 540 00:43:03,460 --> 00:43:03,600 Okay. 541 00:43:04,720 --> 00:43:09,860 Let's do missing models. 542 00:43:10,100 --> 00:43:11,120 So I can download a model. 543 00:43:11,620 --> 00:43:13,000 Let's do this really small one 544 00:43:13,100 --> 00:43:14,240 just 'cause I know it'll download quickly. 545 00:43:15,100 --> 00:43:17,160 Again, a lot of these tools make it so much easier. 546 00:43:17,800 --> 00:43:18,760 If you're just joining the stream, 547 00:43:19,400 --> 00:43:21,400 you can hop into Hugging Face 548 00:43:21,800 --> 00:43:23,740 and you can see all the models that are available. 549 00:43:24,020 --> 00:43:26,980 This is the most popular place for people to go do models, 550 00:43:27,260 --> 00:43:29,420 but it can be very overwhelming because there are so many. 551 00:43:29,860 --> 00:43:30,900 There's almost 2 million models. 552 00:43:31,840 --> 00:43:33,919 So we're gonna use these tools 553 00:43:33,920 --> 00:43:35,320 that kind of curate some of them for us. 554 00:43:35,460 --> 00:43:36,580 All right, so now we've got it. 555 00:43:37,200 --> 00:43:39,180 Hoof, look at how busy this is. 556 00:43:39,760 --> 00:43:40,780 This is way more complicated, 557 00:43:42,160 --> 00:43:43,200 but it's really powerful too. 558 00:43:43,960 --> 00:43:46,060 So let's just hit run with the default prompt 559 00:43:46,080 --> 00:43:46,700 that it gave us. 560 00:43:47,020 --> 00:43:48,280 Prompt execution failed. 561 00:43:49,600 --> 00:43:50,740 Value not in your list. 562 00:43:57,240 --> 00:43:57,640 Okay. 563 00:43:59,520 --> 00:44:02,760 Tutorial, well, there's stuff I could read through here, 564 00:44:02,940 --> 00:44:05,000 but I don't wanna do it right now. 565 00:44:05,800 --> 00:44:07,700 Just see if there's anything quickly jumping out to me. 566 00:44:09,700 --> 00:44:11,220 Looks like the model's loaded there. 567 00:44:12,220 --> 00:44:13,160 I probably need more models. 568 00:44:17,520 --> 00:44:18,320 Another question. 569 00:44:18,400 --> 00:44:18,780 Hey, don't worry. 570 00:44:19,660 --> 00:44:20,440 Chat all you want to. 571 00:44:20,920 --> 00:44:22,700 I'll ignore the questions if you're being too chatty, 572 00:44:22,900 --> 00:44:23,300 but you're good. 573 00:44:24,200 --> 00:44:29,140 The question is, can Comfy UI load fine-tune models? 574 00:44:30,060 --> 00:44:30,860 Let's see. 575 00:44:33,919 --> 00:44:36,700 I think ComfUI can load any model that you want to. 576 00:44:40,279 --> 00:44:40,640 Right? 577 00:44:40,840 --> 00:44:44,000 So they have all these templates here for doing different kinds of things. 578 00:44:44,510 --> 00:44:46,780 But when you go in here, these are all your models that you have. 579 00:44:47,599 --> 00:44:50,540 And you can get any model and throw it in here. 580 00:44:51,640 --> 00:44:55,280 Text encoders, diffusion models, Quinn image is a diffusion model. 581 00:44:55,440 --> 00:44:56,120 That's why it's in there. 582 00:44:57,500 --> 00:45:02,180 So all you do is you get your model and you're going to stick it in one of these folders. 583 00:45:02,400 --> 00:45:06,440 So this is Comfy UI's folder for their models and you put it in here. 584 00:45:06,780 --> 00:45:07,800 It should be able to handle it. 585 00:45:08,140 --> 00:45:14,180 I can't guarantee it, but you can also go look on their website, Comfy UI. 586 00:45:14,980 --> 00:45:15,900 Let's ask Maple. 587 00:45:18,720 --> 00:45:23,520 Can Comfy UI load fine-tuned models? 588 00:45:23,620 --> 00:45:24,920 I'm just going to ask your question directly. 589 00:45:26,400 --> 00:45:33,140 And let's change it over to GPT. 590 00:45:35,220 --> 00:45:35,620 Yes. 591 00:45:36,600 --> 00:45:37,820 Stable diffusion checkpoints. 592 00:45:39,280 --> 00:45:39,880 It can. 593 00:45:40,440 --> 00:45:40,920 All right. 594 00:45:41,460 --> 00:45:44,700 So here's more details for you. 595 00:45:45,740 --> 00:45:47,240 You can read through all that if you want to. 596 00:45:48,520 --> 00:45:50,480 But the short answer is yes, if we trust AI. 597 00:45:50,760 --> 00:45:51,880 Obviously, we have to verify ourselves. 598 00:45:52,600 --> 00:45:55,200 But it looks like promising that you can do it. 599 00:45:55,760 --> 00:45:56,740 So lots of info there. 600 00:45:57,660 --> 00:45:58,120 All right. 601 00:45:58,900 --> 00:46:00,300 Last one to show you is Whisper. 602 00:46:00,820 --> 00:46:02,020 This one might be a little difficult. 603 00:46:02,550 --> 00:46:05,200 I'm going to quit comfy to free up some RAM. 604 00:46:06,300 --> 00:46:09,980 This one might be slightly difficult only because I'm using audio here on the stream. 605 00:46:10,630 --> 00:46:14,620 So I don't know how well this will work, but let's, you can do lots of things in here. 606 00:46:14,700 --> 00:46:16,320 So you can record a meeting that you're on, right? 607 00:46:16,370 --> 00:46:18,920 A lot of people love to have AI join a video call. 608 00:46:19,600 --> 00:46:24,739 It drives me crazy when it's some, this third-party cloud service, because now I know that all 609 00:46:24,740 --> 00:46:29,980 my words are being recorded on some random server that could be susceptible to a data breach or sold 610 00:46:30,010 --> 00:46:40,100 to advertisers or shared with who knows who. So, I'll just be aware of that, right? But you can 611 00:46:40,160 --> 00:46:44,320 record it locally. Why use a third-party server when you can just download an app like MacWhisper, 612 00:46:45,739 --> 00:46:49,940 although verify that this app is not transmitting stuff. You can use tools like Little Snitch and 613 00:46:49,940 --> 00:46:55,000 verify that it's not sending data out to somebody else. I would love to find something like Mac 614 00:46:55,200 --> 00:46:59,760 Whisper that is open source, but this has just been like the easiest thing to use so far. So 615 00:46:59,860 --> 00:47:07,660 that's what I'm using. But yeah, so I can record a meeting. Another thing that I use it for is when 616 00:47:07,740 --> 00:47:13,640 I'm done recording this show is I just drag the audio file into here and it creates a transcript 617 00:47:13,840 --> 00:47:19,039 for me. So that's a, that's a good one. There is this transcribed podcast thing, but what it does 618 00:47:19,040 --> 00:47:25,760 is, let me show you a previous one. Do I have any saved? I usually don't save them. 619 00:47:27,100 --> 00:47:32,200 So, okay. Well, here it's taking one of my previous shows, the MP3s, and it's going to 620 00:47:32,280 --> 00:47:34,820 load it. So, you can see actually what it does here. We're not going to wait for this whole 621 00:47:34,940 --> 00:47:38,740 transcription, but you'll just get the gist as it starts. So, it's currently loading the model 622 00:47:39,220 --> 00:47:46,079 into memory, and now it's got it loaded, and it is starting to transcribe. It's probably going to 623 00:47:46,060 --> 00:47:50,700 take a while because I'm live on the stream. But you can see it's starting to show if there were 624 00:47:50,800 --> 00:47:57,180 multiple speakers, I can go in here and I can add multiple people to this. So Marks is one of the 625 00:47:57,280 --> 00:48:04,740 speakers. Return. Let's say that we had Sally as a speaker, right? So then I can go through and 626 00:48:04,800 --> 00:48:11,799 start. I could like assign them. It would actually detect. So it can detect two different voices 627 00:48:11,800 --> 00:48:16,580 then it will make its best guess and just boom it'll be marks Sally marks 628 00:48:16,920 --> 00:48:20,680 Sally and if for some reason I got those flipped then I just simply change the 629 00:48:20,780 --> 00:48:25,040 names here or I drag them and it'll reverse them all so it's pretty awesome 630 00:48:25,200 --> 00:48:32,120 I love that aspect of it let's delete these here's my transcript as it builds 631 00:48:32,260 --> 00:48:37,960 it here are the segments and then when I'm ready to export this I just let me 632 00:48:37,960 --> 00:48:43,560 back here. I usually do it this way. Export. I'll export it as an SRT file. That's the one I tend 633 00:48:43,570 --> 00:48:49,200 to use. But there's all sorts of options here that you can go with. So pretty sweet. So I 634 00:48:49,680 --> 00:48:54,740 export the SRT and then I just stick that into my RSS feed along with my podcast. 635 00:48:57,760 --> 00:48:58,540 Let's cancel this. 636 00:49:01,960 --> 00:49:07,940 Yes. And so that's also another great comment here from Docstacks. Transcripts from 637 00:49:07,940 --> 00:49:12,540 whisper to summary from LLM into timestamps for your YouTube or X description is a huge time 638 00:49:12,720 --> 00:49:17,320 saver. Totally agree. In fact, I have a massive prompt that I wrote for Freedom Tech Weekend 639 00:49:17,440 --> 00:49:24,240 where I say, you know, here is the transcript of my episode. I want you to output all of these 640 00:49:24,440 --> 00:49:30,560 things, chapters, a show description, give me some tags that I should put into YouTube for this. 641 00:49:31,140 --> 00:49:36,520 I have it generate an image prompt. It doesn't generate the image itself, but I say like 642 00:49:36,520 --> 00:49:41,740 create a prompt that I can give to an image gen AI for making a nice YouTube thumbnail for this, 643 00:49:42,660 --> 00:49:46,800 all sorts of things. So I've got like this eight step process that instead of me doing it myself, 644 00:49:47,030 --> 00:49:52,320 I just take the transcript, paste it in there, and it builds all that for me. And I can do that 645 00:49:52,370 --> 00:49:58,180 all in Maple or locally. It's awesome. One other thing you can do with this that is really helpful 646 00:49:59,000 --> 00:50:05,820 is you can have it do dictation. So like this little screenshot here, this will explain it all. 647 00:50:05,920 --> 00:50:14,260 Most computers do dictation built in like Mac and Windows and stuff, but you're using the like the built in stuff which might go back to their servers. 648 00:50:14,960 --> 00:50:19,500 So if you want to keep a local, you can set up dictation here and then any text box on your screen. 649 00:50:19,940 --> 00:50:32,620 You can just hit a button on your keyboard and the microphone will start and you can start talking to your computer and then it will dictate it for you and process it 100% locally here on your computer using Mac Whisper or some other tool. 650 00:50:32,920 --> 00:50:38,080 I think one is called super whisper if I remember correctly. This is really powerful, especially 651 00:50:38,310 --> 00:50:43,460 when it comes to writing prompts itself. When I'm talking to AI, I can bang it on the keyboard. 652 00:50:43,750 --> 00:50:47,380 Some people prefer that. Sometimes I do. But a lot of times I just hit the microphone button 653 00:50:47,570 --> 00:50:53,180 and dictate to AI and I can speak more like a human because AIs are trained to think like a 654 00:50:53,240 --> 00:51:00,880 human. So when you talk to it in a more familiar way, a more conversational way, then you tend to 655 00:51:00,760 --> 00:51:06,240 to get better results. So dictation is awesome. You can set it up to any key that you want. You 656 00:51:06,250 --> 00:51:15,080 can set up a custom one if you want to. And then what was the thing? Let's try this. I don't know 657 00:51:15,080 --> 00:51:19,880 if it's going to work. Hello, testing, testing, testing. Yeah, it's not working right now because 658 00:51:19,990 --> 00:51:26,020 my microphone is going into the stream. And so it can't do it here. But if I was not streaming 659 00:51:26,090 --> 00:51:28,920 right now with my microphone, then what I just said would show up here in this box, 660 00:51:29,600 --> 00:51:30,460 Just as the test. 661 00:51:30,640 --> 00:51:32,820 And then you can use it anywhere on your operating system. 662 00:51:33,000 --> 00:51:36,840 It'll go into any, it goes into the main level of the operating system. 663 00:51:37,520 --> 00:51:38,440 So cool stuff. 664 00:51:39,160 --> 00:51:39,360 All right. 665 00:51:39,440 --> 00:51:40,600 I think we need to wrap it up here. 666 00:51:40,680 --> 00:51:41,960 We've been going for almost an hour. 667 00:51:42,400 --> 00:51:43,440 We have over 700 people. 668 00:51:43,640 --> 00:51:44,240 Welcome everybody. 669 00:51:44,860 --> 00:51:46,220 Hope you enjoyed today's show. 670 00:51:47,040 --> 00:51:48,740 But we went into a lot today. 671 00:51:49,460 --> 00:51:53,480 Gave you a really good primer in how to do your own AI. 672 00:51:54,340 --> 00:51:55,480 Just real quick. 673 00:51:55,680 --> 00:51:57,560 Like LM Studio is great. 674 00:51:58,740 --> 00:52:02,920 Comfy UI, I just tried out for the first time today for doing image gen and video gen. 675 00:52:04,020 --> 00:52:07,720 And then you've got Whisper for doing audio stuff. 676 00:52:08,160 --> 00:52:10,460 Hugging Face is a great place in general to find models. 677 00:52:11,270 --> 00:52:16,320 Then you have Quen Code, which we put into LM Studio and wrote some stuff for ourselves. 678 00:52:17,320 --> 00:52:22,700 If you want to delete your stuff out of ChatGPT, we can go to OpenAI and request deletion. 679 00:52:23,410 --> 00:52:26,220 And then there are all sorts of blog posts, just like this one that was shared, 680 00:52:26,520 --> 00:52:31,300 for building your own home server if you want something more beefy to chain together multiple 681 00:52:31,520 --> 00:52:38,480 computers because if we want to run these large models for example maple maple runs the new gpt 682 00:52:38,800 --> 00:52:46,740 os 120 oss 120b we also run deep seek the largest deep seek model out there 671b it's massive it 683 00:52:46,800 --> 00:52:51,800 requires huge computing power huge memory to run it you're not gonna be able to do this on your 684 00:52:51,800 --> 00:52:57,480 laptop. So if you want to do bigger things, you can chain them together. All right, let me end 685 00:52:57,530 --> 00:53:03,420 the screen share and let's do a proper send off. So thank you everybody for joining today. Really 686 00:53:03,520 --> 00:53:08,080 appreciate you being here on Freedom Tech Weekend. I feel like the last couple were kind of short and 687 00:53:08,140 --> 00:53:12,360 kind of lame, to be honest. I was traveling and there was a lot going on, especially when you're 688 00:53:12,440 --> 00:53:18,640 doing a startup. Startup, traveling, family, life gets busy, right? No excuses. Let's just, let's do 689 00:53:18,640 --> 00:53:24,500 do the right thing. So I wanted to bring you a much more substantive show today. I hope that I 690 00:53:24,560 --> 00:53:29,380 gave you some things that you can get excited about and try out this weekend. Try out any of 691 00:53:29,380 --> 00:53:33,960 the tools that we did. If you have other ones like ping me, ping me on socials. I'm on X, 692 00:53:34,180 --> 00:53:39,760 I'm on Noster, I'm on YouTube. Hit me up and let me know which ones you like using. 693 00:53:40,739 --> 00:53:45,000 And let's keep the conversation going. And then because I would love to learn from you as well. 694 00:53:45,360 --> 00:53:48,740 And then also get your own private encrypted cloud AI. 695 00:53:48,870 --> 00:53:53,780 Go to try maple.ai, get your free account, upgrade to pro when you need to do more. 696 00:53:54,530 --> 00:53:58,060 And we will talk to you next week on freedom tech weekend. 697 00:53:58,650 --> 00:53:59,680 Hope you all have a great weekend. 698 00:54:00,250 --> 00:54:00,380 Later. 699 00:54:00,520 --> 00:54:00,540 Thank you.