I tried to fine-tune a model to text like me and it failed

The source code and reproducible recipe are on GitHub. The repository includes the scripts and technical documentation; the private dataset and trained weights are not included.

TL;DR

I thought small local models had become good enough that I could fine-tune one on my WhatsApp chats and get it to text like me. I tried LoRA, richer training data, memory, thinking, a separate thinker and texter, and weight edits. Some of the numbers improved. The actual conversations were still disappointing, and I never put the model in my girlfriend's hands.

DeepSeek was really good at understanding the context, but the local texting layer still gave me verbose, unconvincing replies. I came away thinking that fine-tuning is much more task-specific than I had understood, and that memory, intent and having something to say are all separate things to get right. Next, I want to try a larger model and different architectures.

Why I tried this

I had heard about Liquid AI's LFM3-1.7B, and what surprised me was how good these small local models had become. I tried running local models on my laptop, and they were surprisingly good.

That made me think that there is a future where everybody has their own fine-tuned LLM running permanently on their laptop. A model that knows them and can do things in the background. I wanted to get my hands dirty and see how far I could take that.

So I decided to fine-tune a model on my WhatsApp chats. The purpose was to get it to behave more like me.

When I say “behave more like me,” I mean something quite specific. If I put this model in front of my girlfriend to chat, she should not notice the difference. That was the objective I was working towards. I wanted it to run in the background of my own system as well.

Now, this is definitely a hard task. The model needs the immediate context: what has just happened, what we are talking about, and what mood the conversation is in. It also needs a general understanding of both of us, our personalities and how we usually respond to each other. Then there is the longer history, including the things we have been through and the inside jokes that make no sense outside that context.

Even with all of that, it still has to understand the intent behind her latest text, decide how I would respond, and actually speak like me. I went into it thinking I could fine-tune a model on my replies and see how far it got.

Over the next few days, I tried a larger model, a smaller model, thinking, no thinking, one model to think and another to text, and eventually editing the weights. Some of the numbers improved quite a lot. But honestly, using it was disappointing throughout.

I never put it in my girlfriend's hands. It wasn't good enough. I tried chatting with it, tried fine-tuning it again, and it still didn't work the way I wanted.

First, what was I training on?

The first thing I did was export our WhatsApp chats. The initial export had about 161,000 messages, going back to December 2024. There were roughly equal numbers of messages from each of us.

I needed to turn this into examples the model could train on. For each of my replies, the input was the conversation leading up to it, including both her messages and my earlier messages. The output was my next reply. I wanted the model to learn to respond as me, so only my reply contributed to the training loss.

This sounds straightforward, but a WhatsApp export is a messy thing. There are edited-message markers, pasted text, stickers, photos, voice notes, and messages whose meaning depends entirely on something that isn't in the text export.

The first version cleaned the text and simplified the media. I split the conversations into sessions at gaps of three hours. The training and validation sets used earlier sessions, and the test set used sessions starting from 18 September 2026. That gave me about 38,000 training examples in the first version.

The test was basically: given this conversation, how well does the model predict what I actually said next? That was useful, but I would eventually find out how much it left out.

I actually started with Gemma

Although LFM was what got me interested, the first model I actually tried fine-tuning was Gemma. I was using MLX and MLX-LM on my Mac, which has 16 GB of memory. The Gemma model was a 12-billion-parameter model, quantized to four bits, with a LoRA adapter.

LoRA, introduced by Edward Hu and colleagues, lets you train a relatively small set of additional parameters while keeping the original model weights frozen. It made the experiment possible on my system. It didn't mean those additional parameters were a complete representation of my personality. They were an adjustment learned from my conversations.

There were a few bugs before I even got to evaluating the model properly. The training mask was skipping the first few tokens of my replies. Two libraries had different defaults for whether thinking was enabled. The loss was also being calculated on a padding token beyond the actual sequence.

The annoying part is that these things could run without an obvious error. The model was training, the numbers were coming out, and the setup was still wrong.

The early Gemma run did learn to write shorter, more casual replies. But it felt cold. I was also working within the memory limits of the Mac, so I had cut down the context to make training manageable.

Then I moved to the 1.7B model

I switched to LFM3-1.7B. A model this size was much easier to fit on my system. I could use the BF16 weights rather than a four-bit version, and train it with LoRA through MLX.

The first full LFM run took about six hours. The validation loss kept going down. On a sample of 500 test replies, the score improved from 1.858 to 1.413 bits per character. Lower is better here: it means the model assigned more probability to the replies I had actually written.

It also learned some visible parts of how I text. The original model's median reply was 116.5 characters. After fine-tuning, it was 31 characters. My actual replies in that sample had a median of 30.5 characters. The frequency of sending multiple texts in a turn also became much closer to mine.

So LoRA was doing something. The model was learning how I speak.

Then I actually chatted with it, and the context was off. It forgot things, drifted away from the conversation, and sometimes confused who was talking about whom. If I said I was tired, it could start responding as though it was the one who was tired.

This was where the experiment became much more interesting, and much more frustrating. The training metrics looked better. The conversation still didn't work.

To show what this looks like, I later ran made-up conversations against the saved local models. I'm not sharing my original WhatsApp conversations here. The situations and incoming messages in these examples are synthetic; the model replies are actual, unedited outputs. They illustrate the behavior, rather than measuring how often each failure happens.

Here, the model writes short messages, but gets the person wrong. That is exactly the kind of thing that makes the conversation fall apart.

The thinking blocks were one problem

The LFM checkpoint supported a thinking mode, but I had trained it with thinking disabled. In its chat format, that meant an empty think block before my reply.

When I tried turning thinking back on, the fine-tuned model didn't reliably produce a final reply. In one set of eight probes, all eight final replies were empty.

I hadn't intended to train that behavior. I wanted a model that could think about the conversation and then reply like me. I had only supervised the reply, using a format that explicitly skipped thinking.

My first reaction was that I had made the model lose its intelligence. Looking back, that is too broad a claim. What I could see was that its thinking-mode behavior had broken, and its conversation was weak. I hadn't run the kind of evaluation that would tell me how much of its general ability had changed.

There were other problems as well. The first LFM run had only a short window of context. Sessions stopped at three-hour gaps. A repetition penalty was discouraging the model from using words that had appeared in the conversation, including words it might actually need to stay on topic.

So there were multiple things going wrong at the same time.

I had removed some of the conversation while cleaning it

The next thing I did was rebuild the data with media included.

A sticker can carry an entire response. It might mean “sorry,” or affection, or a joke that only the two of us understand. If I turn all of those into the same media placeholder, I have removed the meaning and kept the surrounding replies. The model is then trying to learn from a conversation with pieces missing.

I used a local vision-language model, Liquid AI's LFM3-VL-2B, to describe 2,815 media items. Then I manually labelled the meanings of the most common stickers. The top 100 covered about 88% of the sticker sends.

What the sticker looked like and what it meant between us were different things. A model could describe what a sticker looked like. It couldn't automatically know what that sticker meant between us.

There was also intimacy. I didn't want to clean that out of the data and then expect the model to sound like me in a relationship. That is part of the conversation as well. I put effort into keeping that context in the training data.

Honestly, that didn't work either. Keeping the content was one step. Getting the model to respond in a way that felt like us was much harder, and I didn't get there.

Even the descriptions needed checking. An early prompt made the vision model label ordinary photos as intimate. I updated the system prompt to define the category more clearly, and that fixed the false labels in the small set we rechecked.

That part worked really well for me after the prompt change. The model was fast, and it was doing all of this locally on my system. Big fan of Liquid AI for that.

I increased the context to as many as 40 turns, within a 2,048-token limit, and kept captions and more specific media tags. This was a much more complete version of the conversation than the one I had started with.

Then I tried to give it memory

Recent context still doesn't tell you everything about a relationship. A conversation today can depend on something from months ago.

I had about 24 calendar months of chats, so I started making month-level notes. Then I tried to sequentially compress those into a memory that carried forward. Each month would update what the model knew, and a short card, targeted at about 150 words, would go into the prompt.

For a training example in a particular month, I used the previous month's card. The idea was to give it the background it could have known at that point.

This also failed in ways I hadn't expected. Some of the memory merges became copies of earlier months instead of incorporating the new month. Some dates were wrong. Later, there was a possibility that memories about my work on AI models were pushing the clone towards talking like an assistant.

I had added memory, but I still didn't know if it was remembering the right things. The summary itself could be wrong.

And there was still the separate problem of using what was already in the recent conversation. In another dummy chat, model B acknowledged a change from tea to coffee, then answered with the old drink.

This is a failure to use recent context. It doesn't tell me whether the monthly memory card would have been right.

One model to think, another to text

Once I had the fuller data and memory, I generated short explanations for my replies. I tried local generation, a GPU VM, and eventually DeepSeek through a hosted API.

The stronger model was much better at producing a plausible explanation of the intent behind a reply. That made me think: what if the small model only has to handle the texting?

I ended up training four variants on the H100:

Model	What it learned
A	Write my reply directly
B	Write a short thought, then my reply
C	Take a supplied thought and write my reply
D	Write only the thought

These runs used 13,831 training examples, with the same basic LoRA settings, and took roughly 16 minutes each. The GPU made the actual training much faster than the Mac. Getting everything set up and evaluated was still a separate part of the work.

The loss values in these four panels aren't a ranking. The models were learning different outputs, so their losses mean different things.

On the recorded reply-prediction tests, A was still the strongest. It scored about 1.340 bits per character. B, generating its own thought before the reply, was reported at about 1.473. B thinking and C texting scored about 1.460.

There was an important problem in how I had made the explanations. The teacher model had seen my actual reply before explaining it. At runtime, the thinker didn't have that reply. So C was trained on explanations produced with an advantage that its runtime planner didn't have.

I was asking one model to explain an answer it could already see, and then expecting another model to produce equally useful plans before knowing the answer. Those are different tasks. The explanations could be plausible without being how I had actually thought. That was another learning for me. Next time, I want to generate the training plans using only the information the planner will actually have when it is replying.

DeepSeek understood it. The texter still didn't work

The strongest part of this whole experiment was DeepSeek's reasoning. It was really good at understanding the context and coming up with a plausible way I would respond.

I manually reviewed roughly 50 to 60 samples and marked the reasoning as correct, partially correct or incorrect. Roughly 90% felt correct to me. That is my own judgment on a small sample. It means I agreed with the reasoning in those examples. It doesn't mean the finished conversation was 90% convincing.

The combined conversation did improve. It was better at understanding what was happening and what kind of response made sense. But the texting engine was still not good.

It kept turning a strong plan into something like five messages, all the time. It became very, very verbose. I could read the reasoning and think, yes, that is what I would want to do here. Then I would read the actual messages and they still didn't feel like me.

That was frustrating because the idea made sense to me. Let the stronger model understand the conversation, and let the smaller model handle how I speak. The stronger model was doing its part much better. The texting layer was still failing.

I also tested C separately with a hand-written plan, using only the context available at each turn. That is a controlled check of the local texter, not a rerun of the hosted DeepSeek pipeline. Even with the plan supplied, it split a simple meeting confirmation into four separate lines, including an unnecessary follow-up. That is a small example of the texting layer doing more than I wanted.

This doesn't reproduce the whole hosted pipeline or establish how often it becomes verbose. But even with a correct plan, I still couldn't trust the final text.

In the Hinglish examples, the conversation could go completely off track. Here is another made-up conversation I ran through A. I told it that I was tired after an exam. A few messages later, it was asking about my office, and when I asked where and when to meet, it kept talking about the office.

It had picked up words like “haa” and “bhai,” but the replies didn't make sense in the conversation. That was the frustrating part: familiar language, without the understanding I needed.

I also tried editing the weights

After all of this, I became interested in weight-space programming. The original post I would recommend reading is Sparks of Weight-Space Programming, by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. They explored having a frontier model inspect and edit neural-network weights without gradient training. The possibility of changing behavior that way is exciting to me.

The first edit we tried was a blend between the original model and the fine-tuned model. It retained 25% of the fine-tuning change and moved the weights 75% back towards the base model.

That doesn't mean it behaves like the original model 75% of the time. It is one new set of weights, somewhere between the two checkpoints.

On a small set of eight synthetic tests, the edited checkpoint improved from five correct answers to eight. Then I tried it in conversation and got generic assistant language and invented AI-model context.

In a new dummy chat, the blended model also invented a sick family member while answering a simple question about its day. The other person hadn't supplied that fact.

We also tried restoring the original convolution-module weights while keeping the rest of the fine-tuned model. That produced another experimental checkpoint, but it didn't establish a reliable improvement in conversation either.

I still find the idea interesting. What I tried here was blending and restoring weights. It wasn't evidence that we could directly write my personality or all the missing conversational ability into them.

What was I actually measuring?

I think this is one of the biggest problems in the whole experiment.

The loss measures how well the model predicts my actual reply given the context. That is useful. But there could be several good replies to the same message. The one I happened to send is only one of them.

A model can get closer to my vocabulary, message length and common replies while still failing to understand the moment. It can also give a sensible reply that uses completely different words from the historical one.

So I don't think the loss was meaningless. I think I was asking it to tell me more than it could. I needed separate evaluations for context, speaker identity, intent, style, memory and the ability to sustain a conversation over several turns.

Even a small synthetic test that asks who owns which fact can go well while the actual chat feels wrong. I saw that with the weight edits.

The harder thing is having something to say

Another thing I felt was missing was intentionality.

When I talk to someone, I might want to reassure them, tease them, ask about something from yesterday, or tell them what happened to me. I have things going on in my own life that I bring into the conversation.

The system I built mostly waited for a message and generated a response. It had a slice of conversation and a compressed memory card. It didn't have its own stream of events to talk about, or a clear mechanism for deciding when to start a topic.

A system could be built with those things. But I hadn't built them, and fine-tuning my replies wasn't going to create them automatically.

I still believe there is a version of this that works much better. Somewhere between a 1.7B model and a much larger hosted model, there may be a useful combination of conversational judgment, my way of speaking, and something I can run locally. I don't know where that point is yet.

What I took away from it

I think the biggest learning for me was how task-specific fine-tuning actually is. Picking LoRA was just the beginning. I still had to decide what the examples meant, what context the model needed, what it should learn to produce, and how I would know whether it was getting better.

Claude did most of the implementation with me. It also made mistakes and gave advice that needed checking. I had expected it to catch more of these problems before I had trained the first model. I still needed enough understanding of the task to know what to train, what to check, and what the results actually meant.

My experience with inference changed as well. Renting a GPU, installing vLLM and keeping the workflow running took more effort than I expected. For the reasoning-generation workload I had, using a hosted API was much easier.

Running it myself also felt much more expensive for this workload because I wasn't using the GPU at full capacity. I was paying for the rented time while setting things up, dealing with failures and leaving compute underused. I don't have a clean cost-per-token comparison, but it made it difficult to run economically. Training and inference were separate problems I could make separate decisions about.

Earlier, I thought fine-tuning and inference were so closely connected that doing one would mean doing the other as well. I don't think that is the right way to look at it anymore. Fine-tuning already has difficult problems of its own: understanding the task, preparing the data, choosing what to train and figuring out whether it worked. Inference has its own difficult problems. For this experiment, I could use somebody else's inference service and still have plenty of work left to do on the fine-tuning.

Next, I plan to try a larger model and explore different architectures. I still think there may be a useful point between the 1.7B model I used and the much larger model that understood the conversation well.

But I also want to carry forward the other things this exposed. Sounding like me, understanding the conversation, remembering the right things and having something to say are all parts of the task. A larger model might help. I will still need better data and a way to evaluate each of those things.

I don't want to make this sound like a successful clone with a few rough edges. In the end, it didn't work. Some parts improved, and DeepSeek's reasoning was good, but I still didn't have a model I could put in front of her.

I still believe there is a version of this that can be much better. I want to see if a larger model and a different architecture can get me there.

Tools and references

The methods and models here come from the people building them. The experiment results and the judgment that it didn't work are mine.

• Liquid AI's LFM3-1.7B and LFM3-VL-2B.
• MLX, MLX-LM and MLX-VLM for running and training models locally.
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models.
• Takehi, Teoh, Zhang and Khattab, Sparks of Weight-Space Programming. This inspired the exploration; my weight blending and module restoration were narrower experiments.
• vLLM, used in the rented-GPU workflow.
• Flirting with decision models, a reference for writing up a personal model experiment.