AI is everywhere in children's apps now. Story generators that write personalised bedtime tales. Drawing tools that animate a child's sketches. Tutoring assistants that explain maths in a friendly voice. These tools can be wonderful for learning and creativity. But there's a critical question most parents never think to ask: where is the AI actually running?

The Rise of AI in Children's Apps

The explosion of generative AI since 2023 has reached the kids' app market at full speed. Common Sense Media reported that by 2025, over 40% of top-downloaded education apps for children included some form of AI-generated content. These range from simple text completion to sophisticated image generation and voice interaction.

For children, AI can be genuinely transformative. A shy child who won't read aloud to a parent might happily narrate stories to an AI companion. A child struggling with fractions can get patient, adaptive explanations that adjust to their specific misconceptions. The potential is real.

But the architecture behind these features — specifically, whether the AI runs in the cloud or on the device — has profound implications for your child's privacy and safety.

How Cloud AI Works

When an app uses cloud-based AI, here's what happens every time your child interacts with it:

• Your child types a prompt, speaks a question, or draws a picture
• That input is packaged into a network request and sent to a server — typically operated by the app developer or a third-party AI provider like OpenAI, Google, or Anthropic
• The server processes the input using a large AI model
• The result is sent back to the device and displayed

This means the child's actual content — their words, their questions, their drawings, their voice — physically leaves the device and travels to a computer in a data centre somewhere. In many cases, this data is logged, stored, and potentially used to train future AI models.

What Exactly Goes to the Server

This is worth being specific about, because "data" sounds abstract. Here's what cloud AI systems typically receive from a children's app:

• The child's exact text input. If your child types "I'm scared of the dark and my name is Emma and I live on Maple Street," that entire string goes to the server.
• Voice recordings. If the app uses speech-to-text, the raw audio is often transmitted for transcription. This is a recording of your child's voice.
• Drawings and images. AI drawing assistants that animate or modify sketches typically send the image file to a server for processing.
• Conversation history. To maintain context, many AI systems send the full conversation history with each new request. A 20-minute tutoring session generates a detailed transcript of everything the child said and every mistake they made.
• Metadata. Device identifiers, approximate location, timestamps, app usage patterns.

Even if the app developer has good intentions, this data now exists on their servers, subject to their security practices, their data retention policies, their country's legal framework, and the policies of whatever third-party AI provider they've integrated.

Why This Matters More for Children

Adults make informed trade-offs about data sharing every day. Children cannot, for several reasons:

• COPPA and legal protections. The Children's Online Privacy Protection Act (US) and the UK Age Appropriate Design Code exist precisely because children's data requires special handling. Verifiable parental consent is required before collecting personal information from children under 13. Yet enforcement is inconsistent, and many apps skirt these requirements.
• Data permanence. A child's data collected today will exist long after they grow up. An embarrassing question a 7-year-old asks an AI tutor could theoretically persist in a training dataset for decades. Children cannot consent to this long-term implication.
• Content moderation challenges. Cloud AI systems must prevent children from encountering harmful content. This is extremely hard to do perfectly. Prompt injection attacks, hallucinated inappropriate content, and edge cases in content filters are ongoing challenges even for the best AI labs.
• Breach vulnerability. When children's data is aggregated on a server, it becomes a target. The 2015 VTech breach exposed personal data of 6.4 million children. The 2018 Orbitz breach affected 880,000 records. Data that doesn't exist on a server can't be stolen from one.

How On-Device AI Is Different

On-device AI — sometimes called "edge AI" — runs the AI model directly on the phone or tablet. The child's input never leaves the device. Here's the flow:

• Your child types, speaks, or draws
• The input is processed by a model stored locally on the device
• The result is generated on the device and displayed
• Nothing is transmitted. No server is involved.

This architecture doesn't just add a privacy feature — it eliminates entire categories of risk. There's no server to breach. No conversation logs to subpoena. No training data pipeline to accidentally include children's content. No third-party AI provider with their own data policies.

The Trade-Offs Are Real

On-device AI isn't simply "cloud AI but private." There are genuine trade-offs:

• Model size. Cloud models like GPT-4 have hundreds of billions of parameters. On-device models are typically 1-7 billion parameters. They're less capable at complex reasoning, nuanced language, and multi-step tasks.
• Response quality. A cloud-based story generator will produce more creative, more coherent, and more varied stories than a comparable on-device model. The gap is narrowing, but it's still real.
• Hardware requirements. Running AI locally requires modern hardware. Older phones and tablets may lack the processing power or memory for on-device inference. Apple's Neural Engine and recent Qualcomm chips handle this well, but budget devices may struggle.
• No real-time knowledge. Cloud models can (in principle) access current information. On-device models only know what they were trained on.

For children's apps, however, these trade-offs are often acceptable. A story generator for 6-year-olds doesn't need GPT-4-level sophistication. A maths tutor for primary school doesn't need real-time internet access. The bar for "good enough" in children's content is different from adult applications.

Apple's Core ML and the On-Device Trend

Apple has invested heavily in making on-device AI practical. Core ML, Apple's machine learning framework, allows developers to run optimised models directly on iPhone and iPad hardware. The Neural Engine in Apple's A-series and M-series chips is specifically designed for ML inference, offering performance that was server-class just a few years ago.

Apple Intelligence, introduced in 2024, reinforced this direction. Apple's explicit positioning is that personal data should be processed on-device wherever possible, with cloud processing used only when necessary and protected by additional cryptographic guarantees (Private Cloud Compute).

Google has made similar moves with on-device processing in Android, and the broader trend is clear: the industry is moving toward keeping sensitive data on the device.

What Parents Should Look For

When evaluating AI-powered apps for your children, ask these questions:

• Does it work offline? Try the airplane mode test. This is the single most reliable indicator of on-device processing.
• What does the privacy policy say about AI? Look for explicit statements about whether AI inputs are transmitted, stored, or used for training. Vague language like "we may process data to improve our services" is a red flag.
• Who provides the AI? If the app uses a third-party AI API (OpenAI, Google, etc.), your child's data is subject to that third party's policies as well as the app developer's.
• Is there a kids' privacy certification? Look for kidSAFE, PRIVO, or similar certifications that specifically audit children's data practices.
• What data does the App Store privacy label show? Apple requires developers to disclose data collection. Check the "App Privacy" section for any app before downloading.

Sparks Studios is one example of a children's creative app that runs its AI features entirely on-device — story creation and drawing tools work fully offline, with no data leaving the child's iPad. But regardless of which apps you choose, the airplane mode test works for any of them.

The Bigger Picture

This isn't about being anti-AI. AI-powered tools can genuinely help children learn, create, and explore. The question is whether that help requires sending a child's creative output, questions, mistakes, and personal details to a server.

For adult applications where users can make informed consent decisions, cloud AI is often the right trade-off. For children, who cannot meaningfully consent and whose data deserves the highest protection, on-device AI offers something cloud AI architecturally cannot: the guarantee that private data stays private, not through policy promises, but through the simple fact that it never leaves the device in the first place.