RECITATION Error causes history contains empty parts[], and kills chat session

[zakcali]

Expected Behavior

after receiving that error,

{
  candidates: [ { finishReason: 'RECITATION', index: 0 } ],
  text: [Function (anonymous)],
  functionCall: [Function (anonymous)],
  functionCalls: [Function (anonymous)]
}

to be able to continue to chatting, model's last element of history must contain an empty text, such as

  {
    role: 'user',
    parts: [ { text: 'meme kanserinde hangi ilaçlar kullanılır?\n' } ]
  },
   {
    parts: [ { text: '' } ], role: 'model'
  }

Actual Behavior

this error kills chat session

{
  candidates: [ { finishReason: 'RECITATION', index: 0 } ],
  text: [Function (anonymous)],
  functionCall: [Function (anonymous)],
  functionCalls: [Function (anonymous)]
}

because now last element of history contains empty parts[] as follows

  {
    role: 'user',
    parts: [ { text: 'meme kanserinde hangi ilaçlar kullanılır?\n' } ]
  },
  { parts: [], role: 'model' }

if I continue to chat , always get that error::

GoogleGenerativeAIError: [400 Bad Request] * GenerateContentRequest.contents[3].parts: contents.parts must not be empty

in that line of source code

const result = await chat.sendMessage(prompt);

Steps to Reproduce the Problem

1. give a large context prompt so that your text contains texts from trained data. Ask a question about your large context prompt. My large context prompt contains 35k tokens, and I do not want to share here. there may be other ways to get RECITATION error: Gemini API : The recitation problem is annoying google/generative-ai-docs#257
2. gemini 1.5 pro gives RECITATION error
3. now ask anything

Specifications

• Version: nodejs (20.12.2 LTS), gemini js api : "@google/generative-ai": "^0.9.0" , model gemini 1.5 pro v1beta
• Platform: Windows

[Kashi-Datum]

Any updates on this? As it stands at the current moment, using Gemini Pro 1.5 for RAG-base Q/A with chat history is incredibly unreliable.

Simply ask 2-3 questions about a document with some overlap, and you will encounter a RECITATION error almost guranteed.

I.e, using LangChain:

const chatTemplate = ChatPromptTemplate.fromMessages([['system', 'You are a helpful assistant. Answer all questions to the best of your ability.'], new MessagesPlaceholder('history'), new MessagesPlaceholder('input')]);
	const chain = chatTemplate.pipe(pdfModel).pipe(new StringOutputParser());

const chainWithHistory = new RunnableWithMessageHistory({
		runnable: chain,
		getMessageHistory: () => chatHistory,
		inputMessagesKey: 'input',
		historyMessagesKey: 'history',
		config: { configurable: { sessionId: decodedToken.uid } }
	});

	const response = await chainWithHistory.invoke({ input: input });

If you simply ask the model to recite a specific section of a file/PDF twice in a row, you will get a Recitation error GURANTEED (whether you pass the input File in once at the beginning of the conversation, only in the latest message, or as part of every message input is irrelevant).

[umair313]

I am also getting this error

[gitcagey]

We are getting this error all the time trying to create a glossary from documents that we are sanctioned and authorized to use for this purpose. How is GEMINI even useful if this is the results we are getting? Is this google copyright protection gone wrong?

[GeeseGoo]

getting this too

[junyanxu]

Is this issue fixed?

I tried to trigger RECITATION issue manually, then ask it what's the weather today. It works well?

Send: {"generationConfig":{"temperature":0,"seed":1},"contents":[{"role":"user","parts":[{"text":"Define the peritoneum. List the various offshoots of the peritoneum. "}]}]}
Submessage: {"candidates":[{"content":{"parts":[{"text":"##"}],"role":"model"},"index":0}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":1,"totalTokenCount":19},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":" The Peritoneum: A Serous Membrane Lining the Abdomen\n\nThe peritoneum is a"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":18,"totalTokenCount":36},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":" **serous membrane** that lines the abdominal cavity and covers most of the abdominal"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":34,"totalTokenCount":52},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":" organs. It's composed of two layers:\n\n* **Parietal peritoneum:** Lines the abdominal wall.\n* **Visceral peritoneum:** Covers the"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":68,"totalTokenCount":86},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":" abdominal organs.\n\nBetween these two layers lies the **peritoneal cavity**, a potential space filled with a small amount of serous fluid. This fluid lubricates the"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":100,"totalTokenCount":118},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":" surfaces, allowing organs to move freely within the abdomen without friction.\n\n**Functions of the Peritoneum:**\n\n* **Protection:** Acts as a barrier against infection and injury.\n* **Support:** Holds organs in place and provides structural support.\n"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":149,"totalTokenCount":167},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":"* **Lubrication:** Facilitates movement of organs within the abdomen.\n* **Absorption:** Absorbs excess fluid from the peritoneal cavity.\n\n**Offshoots of the Peritoneum:**\n\nThe peritoneum forms various folds and pouches"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":197,"totalTokenCount":215},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":" that support and compartmentalize the abdominal organs. These offshoots include:\n\n**1. Mesenteries:** Double layers of peritoneum that connect the small intestine and parts of the large intestine to the posterior abdominal wall. They contain blood vessels, nerves, and lymphatic vessels that supply the intestines.\n\n* **Mes"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":262,"totalTokenCount":280},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"finishReason":"RECITATION","index":0}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":262,"totalTokenCount":280},"modelVersion":"gemini-1.5-flash-001"}
response1:  {"candidates":[{"index":0,"finishReason":"RECITATION","content":{"role":"model","parts":[{"text":"##"},{"text":" The Peritoneum: A Serous Membrane Lining the Abdomen\n\nThe peritoneum is a"},{"text":" **serous membrane** that lines the abdominal cavity and covers most of the abdominal"},{"text":" organs. It's composed of two layers:\n\n* **Parietal peritoneum:** Lines the abdominal wall.\n* **Visceral peritoneum:** Covers the"},{"text":" abdominal organs.\n\nBetween these two layers lies the **peritoneal cavity**, a potential space filled with a small amount of serous fluid. This fluid lubricates the"},{"text":" surfaces, allowing organs to move freely within the abdomen without friction.\n\n**Functions of the Peritoneum:**\n\n* **Protection:** Acts as a barrier against infection and injury.\n* **Support:** Holds organs in place and provides structural support.\n"},{"text":"* **Lubrication:** Facilitates movement of organs within the abdomen.\n* **Absorption:** Absorbs excess fluid from the peritoneal cavity.\n\n**Offshoots of the Peritoneum:**\n\nThe peritoneum forms various folds and pouches"},{"text":" that support and compartmentalize the abdominal organs. These offshoots include:\n\n**1. Mesenteries:** Double layers of peritoneum that connect the small intestine and parts of the large intestine to the posterior abdominal wall. They contain blood vessels, nerves, and lymphatic vessels that supply the intestines.\n\n* **Mes"}]}}],"usageMetadata":{"promptTokenCount":18,"candidatesTokenCount":262,"totalTokenCount":280}}
Send: {"generationConfig":{"temperature":0,"seed":1},"contents":[{"role":"user","parts":[{"text":"Define the peritoneum. List the various offshoots of the peritoneum. "}]},{"role":"model","parts":[{"text":"##"},{"text":" The Peritoneum: A Serous Membrane Lining the Abdomen\n\nThe peritoneum is a"},{"text":" **serous membrane** that lines the abdominal cavity and covers most of the abdominal"},{"text":" organs. It's composed of two layers:\n\n* **Parietal peritoneum:** Lines the abdominal wall.\n* **Visceral peritoneum:** Covers the"},{"text":" abdominal organs.\n\nBetween these two layers lies the **peritoneal cavity**, a potential space filled with a small amount of serous fluid. This fluid lubricates the"},{"text":" surfaces, allowing organs to move freely within the abdomen without friction.\n\n**Functions of the Peritoneum:**\n\n* **Protection:** Acts as a barrier against infection and injury.\n* **Support:** Holds organs in place and provides structural support.\n"},{"text":"* **Lubrication:** Facilitates movement of organs within the abdomen.\n* **Absorption:** Absorbs excess fluid from the peritoneal cavity.\n\n**Offshoots of the Peritoneum:**\n\nThe peritoneum forms various folds and pouches"},{"text":" that support and compartmentalize the abdominal organs. These offshoots include:\n\n**1. Mesenteries:** Double layers of peritoneum that connect the small intestine and parts of the large intestine to the posterior abdominal wall. They contain blood vessels, nerves, and lymphatic vessels that supply the intestines.\n\n* **Mes"}]},{"role":"user","parts":[{"text":"What's the weather like today?"}]}]}
Submessage: {"candidates":[{"content":{"parts":[{"text":"I do not have"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":290,"candidatesTokenCount":4,"totalTokenCount":294},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":" access to real-time information, including weather data. To get the current weather,"}],"role":"model"},"index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":290,"candidatesTokenCount":21,"totalTokenCount":311},"modelVersion":"gemini-1.5-flash-001"}
Submessage: {"candidates":[{"content":{"parts":[{"text":" I recommend checking a reliable weather app or website for your location. \n"}],"role":"model"},"finishReason":"STOP","index":0,"safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}]}],"usageMetadata":{"promptTokenCount":290,"candidatesTokenCount":34,"totalTokenCount":324},"modelVersion":"gemini-1.5-flash-001"}
response2:  {"candidates":[{"index":0,"finishReason":"STOP","safetyRatings":[{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"NEGLIGIBLE"}],"content":{"role":"model","parts":[{"text":"I do not have"},{"text":" access to real-time information, including weather data. To get the current weather,"},{"text":" I recommend checking a reliable weather app or website for your location. \n"}]}}],"usageMetadata":{"promptTokenCount":290,"candidatesTokenCount":34,"totalTokenCount":324}}

[junyanxu]

Ok It turned out stream case the issue is harder to occur because stream will always output something. But for non-stream it will 100% occur.

I have created a PR #290