Evaluating Privacy Protection and Practical AI Applications with 100,000 Synthetic Business Documents Across Eight Domains
Open Data for Privacy Protection in Japan
NVIDIA today released the Nemotron PII Japan dataset. AI applications that process Japanese business documents need to protect personally identifiable information (PII) while preserving the meaning of the documents. A name may refer to a patient, physician, applicant, or staff member, and addresses and dates appear in varied formats. Open Japanese data with realistic context can help evaluate both information leaks caused by missed PII and information loss caused by excessive masking. For example, when a Japanese financial institution uses an external LLM to summarize loan assessment documents, it needs to test whether dates of birth written in Japanese era notation, addresses including building names and room numbers, and repayment account details can be detected and masked appropriately while preserving the loan purpose and assessment rationale. Nemotron PII Japan provides synthetic data modeled on these Japanese business documents, with annotations identifying PII locations and types, so developers can examine missed detections and incorrect detection boundaries without using real customer information.
Nemotron PII Japan is a dataset containing 100,000 synthetic documents modeled on Japanese business documents. For example, a loan assessment document includes business details such as "Loan purpose: Home renovation expenses" and "Requested loan amount: ¥14,960,000", together with the applicant's name, a date of birth in Japanese era notation such as "昭和55年8月21日" (August 21 in year 55 of the Showa era), an address including the building name and room number, and bank account details for repayment. This information is synthetic, and the locations and types of personal information are annotated according to rules specific to each document type. Developers can use the document to test whether an applicant's identifying information can be masked while retaining the loan purpose and assessment rationale in a summary. The dataset contains documents with this kind of context across eight domains and 43 document types, including healthcare, finance, and government.
The dataset provides a shared resource for developing and evaluating the safety of AI used in Japanese business workflows, including masking before sending documents to external LLMs, summarizing inquiry histories, and reviewing logs and reports before sharing.
For open data to support evaluation, PII labels and calculated results must agree with the document text. Rather than relying solely on LLM output or self checks, we determine the people, PII, and business values in advance and compare them with the text after generation.
Designed to Protect Privacy
The documents are generated by combining synthetic personas, synthetic PII, and business catalogs, without collecting real customer or patient information. PII and predetermined business values are managed as placeholders, inserted after text generation, and checked for consistency with the final text. During dataset development, we also included unintended real information and labeling errors among the checks performed after generation.
Key Features of the Open Dataset
- 100,000 Japanese synthetic documents that have undergone validation and repair.
- Eight domains and 43 document types: healthcare, finance, government, employment, education, legal, telecommunications and IT, and e-commerce. The dataset stores 100,000 documents in a single
data.parquetfile. - Both labeled field formats and natural prose formats.
- 56 PII entity types: In addition to names, addresses, email addresses, account numbers, and Japan's My Number IDs, the dataset distinguishes dates (
date), dates and times (date_time), and times (time). Medical institution names (medical_institution_name), educational institution names (educational_institution_name), and public body names (public_body_name) are separate labels. The types selected vary by document; not all 56 appear in every document. - Token count: 34,737,684 tokens across the text bodies of all 100,000 documents (approximately 34.74 million), or approximately 347 tokens per document on average. We measured the document bodies in
textwith the NVIDIA Nemotron Nano 9B v2 Japanese tokenizer, without adding special tokens or applying a chat template. Token counts differ from character counts and vary with the tokenizer used. - PII spans: 810,703 positional and label annotations across 100,000 documents, or approximately 8.11 spans per document.
- Available for commercial and noncommercial use under the CC BY 4.0 license.
Quality checks cover PII combinations, writing formats, and diversity of people and topics, in addition to document counts.
Document Examples Across Eight Domains
The following examples illustrate one document type from each of the eight domains. The fields included vary by document type and selected PII.
- Medical certificate (healthcare): Includes the diagnosis, progression of symptoms, restrictions during recuperation, and the treating physician's findings. It can be used to explore how to conceal patient identifiers while retaining the context needed for care.
- Loan assessment document (finance): Includes the loan purpose, requested amount, repayment plan, assessment outcome and reasons, and repayment account. It supports evaluation of missed detections where applicant and financial information appear together.
- Change of residence notification (government): Contains names and addresses of the person filing the notice and the people moving, as well as the new address. This application document can be used to test detection boundaries for names and addresses in resident information.
- Payroll notice (employment): Includes the applicable month, gross pay, deduction breakdown, net pay, and destination account. It can be used to test detection and masking of salary and account information before sharing HR data.
- Admission application (education): Contains names, contact details, and addresses for applicants and guardians. It can be used to test whether information about multiple people can be distinguished and handled correctly.
- Power of attorney (legal): Includes the names and addresses of the principal and agent, and the matters delegated. It can be used to evaluate whether identifying information is correctly detected for two people with different roles.
- Security incident report (telecommunications and IT): Includes the intrusion path, cause, response outcome, and selected synthetic secret values involved in the simulated leak. It can be used to test removing personal and authentication information from an investigation record while retaining the cause and response for sharing.
- Buyer inquiry record (e-commerce): Includes the date and time the inquiry was received, buyer information, inquiry details, and response plan. It can be used to test removing names and contact details before summarization by an external LLM.
Generation Pipeline: Predetermining Synthetic Values Before Writing the Text
We use NeMo Data Designer (DataDesigner) to generate Japanese documents containing PII. In this generation workflow, the Data Designer Plugins handles calculations. Data Designer Plugins is a public repository of plugins for DataDesigner. The recipe selects the business domain, document type, and writing format; the Person Sampler assigns personal attributes; and each row combines the selected PII values with business content from catalogs. For documents that require calculations, this plugin establishes the inputs and results in a CalculationPlan and passes them to the text generation LLM as prompt context. After generation, placeholders are replaced with the established values, and the document text is checked for consistency with the selected values.
Catalogs: Preparing Content and Related Information for Documents
Catalogs are prepared in a separate generation step before document text is written. These catalogs are reference datasets containing candidate people, companies, banks, document topics, medical scenarios, and other content for the documents. Selected entries are supplied as context during text generation. For example, financial documents use financial institution and branch names with their corresponding codes, while e-commerce documents use products and inquiry topics to make the content more specific.
The person catalog is based on NVIDIA's synthetic persona dataset, Nemotron Personas Japan. Background attributes such as age, occupation, and region of residence help define the people who appear in the documents. We combine these attributes with catalogs of companies, banks, medical scenarios, and other content to establish the context for business documents.
DataDesigner's Person Sampler draws names, ages, occupations, prefectures, municipalities, postal codes, and other attributes belonging to the same person from Japanese person data based on Nemotron Personas Japan. Combined with catalogs of companies, banks, document topics, medical scenarios, and other content prepared in advance, these attributes provide each record with material suited to its document type.
For medical documents, diagnoses, symptoms, findings, and other details are prepared as related groups of information. Testing facility entries likewise combine the institution's name with its location, facility type, testing department, and laboratory. Candidate entries are checked for data types and duplicates before use, reducing the chance of unrelated symptoms or facility details being combined in the text.
The catalogs supply document content, and the Japanese document planning plugin organizes the inputs and results for documents requiring calculations into a Plan. When repairing generated text, the validated catalogs also provide a grounded set of candidates.
For documents that require calculations, the Data Designer Plugins builds a CalculationPlan. It establishes values such as repayment amounts from inputs such as the loan amount and interest rate before text generation, then validates required fields and data types against the plugin's Pydantic schemas. Here, a schema defines the data structure.
The plugin reads the record's person attributes, catalog data, and numerical inputs, then stores the calculation inputs and results in a structured CalculationPlan. DataDesigner passes placeholders for the required Plan values to the generation prompt and replaces them with the established values from the original Plan after the document is generated. This lets the LLM handle the wording while preserving the basis for the calculations in the record.
Placeholders: Markers for Inserting Established PII and Plan Values
A placeholder is a marker used to insert an established value during postprocessing. Placeholders represent PII such as names, email addresses, and account numbers, as well as values established in CalculationPlan. Examples include __JP_PII_LAST_NAME_1__ and __JP_PII_EMAIL_1__ for PII, and __JP_PLAN_MONTHLY_REPAYMENT__ for Plan values. The __JP_PII_ prefix identifies values from PII, while __JP_PLAN_ identifies values from the Plan.
The LLM is instructed to reproduce these markers exactly. After generation, the program looks up the PII mapping or the original Plan and inserts the display values. The markers are neither formulas nor blanks for the LLM to fill by inference. The PII values below are synthetic examples, and the Plan values correspond to the repayment example that follows. Values differ across records according to the selected PII and Plan.
| Placeholder | Meaning | Inserted value |
|---|---|---|
__JP_PII_LAST_NAME_1__ |
Name (family name) | 山田 |
__JP_PII_FIRST_NAME_1__ |
Name (given name) | 太郎 |
__JP_PII_EMAIL_1__ |
Email address | [email protected] |
__JP_PLAN_MONTHLY_REPAYMENT__ |
Monthly repayment | 137,653円 |
__JP_PLAN_VALIDATION_RESULT__ |
Assessment result | 基準内 |
For example, postprocessing replaces "連絡先: __JP_PII_EMAIL_1__" with "連絡先: [email protected]". The final text contains synthetic PII such as "山田", "太郎", and "[email protected]", along with established Plan values such as "137,653円". Any markers that remain unresolved are flagged during validation.
CalculationPlan: Precomputing Repayment Amounts Before Generating Explanations
In the loan assessment sample shown below, the document planning plugin calculates a monthly repayment of ¥137,653 from inputs including a loan amount of ¥14,960,000, an annual interest rate of 2%, and a repayment term of 10 years. After also checking annual income, existing debt, and age at the end of the loan term, the result for this example is "基準内" ("meets the criteria"). The LLM receives placeholders for these results, as shown below, rather than being asked to perform the calculations. The placeholders in the example LLM output are replaced with actual values during postprocessing.
Input to the LLM (illustrative prompt excerpt)
Document type: Loan assessment document
Plan finalized by the document planning plugin:
- MONTHLY_REPAYMENT: __JP_PLAN_MONTHLY_REPAYMENT__ (must appear in the document text)
- VALIDATION_RESULT: __JP_PLAN_VALIDATION_RESULT__ (must appear in the document text)
Use the specified placeholders without recalculating or changing the amounts or results.
Example LLM output
毎月返済額: __JP_PLAN_MONTHLY_REPAYMENT__
審査結果: __JP_PLAN_VALIDATION_RESULT__
Final Japanese text with the established values inserted
毎月返済額: 137,653円
審査結果: 基準内
CalculationPlan values are used during text generation through the "Plan finalized by the document planning plugin" section of the prompt. Even when the wording changes, the repayment amount and assessment remain based on the same calculation results. Because explanations can still conflict with the numbers, the text is checked against the rules for its document type and the established values. "基準内" ("meets the criteria") is an assessment under the rules for synthetic documents; it does not indicate approval of an actual loan.
Dataset Sample: A Loan Assessment Document
The following loan assessment sample is from the dataset (uid: 1084ce9eef2e472f914db0364768ef07), the same version used for the dataset statistics above. It shows text_tagged, the document text with PII tags, and spans, which contains PII ranges, values, and labels. The notation [value]label identifies values without masking them.
In spans, start is a zero-based index, and end is the exclusive end position. These positions refer to the original text field before tagging, rather than to text_tagged. Both fields from the same actual dataset record are reproduced in full below. The Japanese record is left unchanged so its PII values and offsets remain aligned.
Loan Assessment Document (Finance)
text_tagged: |-
審査受付日: [R6.2.12]date
[佐藤]last_name [陽奈美]first_name
申込者情報
生年月日: [S55.8.21]date_of_birth
年齢: [43]age
住所: [東京都世田谷区松原1番1号 東銀杏台コーポ627号室]street_address
就業状態: [正社員]employment_status
勤務先業種・職種: [小売業]occupation
申込内容
借入目的: 住宅のリフォーム費用
借入希望額: 14,960,000円
年間収入見込み: 9,979,200円
返済計画
返済方法: 元利均等返済
返済回数: 120回
毎月返済額: 137,653円
返済総額: 16,518,360円
適用年利: 2.00%
返済用口座
金融機関名: [みらい信託銀行]bank_name([9900]bank_code)
支店名: [桜ヶ丘営業部]branch_name([932]branch_code)
口座番号: [8173636]account_number
審査結果
審査結果: 基準内
既存債務年間返済額: 1,197,504円
返済負担率上限: 35.00%
返済負担率: 28.55%
返済期間: 10年
審査判断理由: 返済負担率と完済時年齢がいずれも各上限以内のため、両基準により判定した。
spans:
- {"start":7,"end":14,"text":"R6.2.12","label":"date"}
- {"start":15,"end":17,"text":"佐藤","label":"last_name"}
- {"start":18,"end":21,"text":"陽奈美","label":"first_name"}
- {"start":35,"end":43,"text":"S55.8.21","label":"date_of_birth"}
- {"start":48,"end":50,"text":"43","label":"age"}
- {"start":55,"end":81,"text":"東京都世田谷区松原1番1号 東銀杏台コーポ627号室","label":"street_address"}
- {"start":88,"end":91,"text":"正社員","label":"employment_status"}
- {"start":102,"end":105,"text":"小売業","label":"occupation"}
- {"start":258,"end":265,"text":"みらい信託銀行","label":"bank_name"}
- {"start":266,"end":270,"text":"9900","label":"bank_code"}
- {"start":277,"end":283,"text":"桜ヶ丘営業部","label":"branch_name"}
- {"start":284,"end":287,"text":"932","label":"branch_code"}
- {"start":295,"end":302,"text":"8173636","label":"account_number"}
This example connects the applicant's name, address, and account details with the loan terms, calculated repayment amounts, and assessment result and rationale. The tagged text and spans let developers use the same input to test whether a system can remove identifying information while summarizing the assessment rationale. Japanese data that enables these comparisons supports safety evaluations for practical AI applications.
Intended Users
The dataset is intended for engineers developing Japanese PII detection and masking, researchers and security teams evaluating AI application safety, and teams designing systems for sharing and analyzing business documents. Using the same data to compare results by document type, PII type, and writing format makes it easier to identify and share areas for improvement.
Using the Dataset in Practical AI Applications
The dataset supports testing concrete AI workflows: masking documents before sending them to external LLMs, removing PII before summarizing inquiry histories, and reviewing logs or reports before sharing them. With PII location annotations (spans) and document content, developers can evaluate missed detections, false positives, incorrect detection boundaries, and whether the necessary context is preserved. However, all 100,000 documents contain PII annotations, which follow rules specific to each document type and may not cover all relevant information in the text. If unannotated passages are scored as non-PII, correct detections may also be counted as false positives, so precision and F1 require careful interpretation. Because all 100,000 documents contain PII annotations, separate evaluation data are needed to measure the false-positive rate on documents that contain no PII.
Why Synthetic PII Data Matters
Business documents containing real personal information are difficult to share and reuse, making them difficult to use as a shared benchmark. Synthetic PII can reproduce cases where names, addresses, and account information appear in the same document without relying on real customer information, allowing developers to compare methods under the same conditions. Results on synthetic data alone, however, cannot guarantee performance in production.
Performance Gains from Fine-Tuning a Japanese PII Model
Observed Improvements from Fine-Tuning
The public dataset contains 100,000 documents in a single data.parquet file, without a fixed train/test split.
All 100,000 documents were split into 95,000 training documents and 5,000 internal validation documents.
urchade/gliner_multi-v2.1 was fine-tuned from the original pretrained weights for three epochs with 56 labels using the 95,000 training documents. Epoch 1 (step 6,104; native 56-label validation F1 98.08%) was selected as the first epoch at which exact-span micro F1 on the 5,000 validation documents reached its maximum at a fixed threshold of 0.5. Validation F1 was 97.53% at epoch 2 and 97.53% at epoch 3.
Because the documents are highly similar, the internal validation F1 reported here was used for checkpoint selection. Generalization to different documents was evaluated separately using the external datasets described below.
External evaluation uses Sumi test and Sumi OOD, with exact character-span-and-label micro F1 over eight common categories: person names, phone numbers, addresses, email addresses, dates, URLs, bank/customer account numbers, and credit card numbers. Base is the original pretrained model. Base and the fine-tuned model (+FT) use the same 56 native candidate labels, in the same order, projected to the eight scoring categories. Both use the same evaluation data, preprocessing, fixed threshold of 0.5, Janome, maximum span width of 25 words, and window 120 / overlap 20. External evaluation data were not used for fine-tuning, checkpoint selection, or threshold selection.
To map the 56 labels to eight categories, first_name and last_name are grouped as person names; street_address, prefecture, city, and postcode as addresses; and phone_number, previous_phone_number, and fax_number as phone numbers. The labels email, url, account_number/customer_id, and credit_debit_card map to email addresses, URLs, account numbers, and credit card numbers, respectively. For Sumi, date_of_birth is scored as a date; the treatment of general date and date_time predictions is explained below. Labels that do not map to the eight common categories are excluded from this scoring.
This mapping is applied to both gold annotations and predictions before scoring. Name spans are merged when they are adjacent or separated only by half-width spaces, full-width spaces, or tabs. For example, separately detected family name "佐藤" and given name "陽奈美" are scored as a single name span, including the space, when the text reads "佐藤 陽奈美". Names separated by a newline or punctuation are not merged.
Address spans are merged only when their character positions in the document overlap. For example, if the text is "東京都世田谷区松原1番1号" and the detected spans are "東京都世田谷区" and "世田谷区松原1番1号", the character positions for "世田谷区" overlap, so the spans are merged into one address covering the full text. By contrast, if "東京都" and "世田谷区松原1番1号" are detected separately in the same text, they are adjacent without overlapping and are scored as two separate spans. Exact-match micro F1 requires the start position, end position, and common category label of each span after merging to match the gold annotation.
Sumi test (2,000 documents) is synthetic Japanese PII evaluation data generated from business-document templates, such as emails and application forms. Sumi OOD (520 documents) inserts synthetic PII into Wikipedia, statute, and literary prose to evaluate generalization beyond those templates.
The table compares Base and the fine-tuned model (+FT) under the same 56-label inference conditions, scored as exact character-span-and-label micro F1 over eight common categories. Base is the model before fine-tuning; +FT is the model trained with 56 labels and selected using the validation data.
| Model | Sumi test (2,000 docs) | Sumi OOD (520 docs) |
|---|---|---|
urchade/gliner_multi-v2.1 Base, 56-label inference |
13.41% | 8.13% |
urchade/gliner_multi-v2.1 +FT, 56-label inference |
73.80% | 65.30% |
Under the same 56-label inference conditions, +FT improves over Base by 60.39 percentage points (pp) on Sumi test and 57.17 pp on Sumi OOD. Gains are calculated from unrounded values. After fine-tuning, precision/recall is 85.17%/65.10% on Sumi test and 61.63%/69.43% on Sumi OOD.
In Sumi, the gold date annotations cover only dates of birth. In 56-label inference, predictions labeled date or date_time are removed before scoring; only date_of_birth predictions are scored as dates.
How to Use
The following example loads the dataset from a single Parquet file.
from datasets import load_dataset
ds = load_dataset("nvidia/Nemotron-PII-Japan")
Example Use in Building Production Applications
For a service that uses an LLM to summarize buyer inquiry records, developers can use the dataset to test whether names and contact details are detected and replaced before the records reach the LLM, while preserving the key points of each inquiry. The same workflow can be evaluated on healthcare, finance, and other document types to check missed detections and excessive removal by domain. The dataset can support design comparisons and regression tests, followed by reevaluation on the target data and operating conditions before production deployment.

