fix(insights): polish insights generation (#730)

* polish insights

* polish insights

* polish insights

* polish insights

* polish insights

* polish insights

* polish insights

* fix pytest error

configs/agent_prompt/brainstorm_idea.yaml

@@ -7,23 +7,26 @@ fewshot_examples:
 - |
   Here is your research background: I am a researcher focused on reinforcement learning (RL), particularly in model-based RL, value-function approximation, and off-policy evaluation. My work aims to connect theoretical properties with empirical performance, addressing issues like error compounding in model-based RL and the limitations of loss functions such as the MuZero loss. One of my key contributions is the development of boundary-invariant analyses for RL algorithms, which provide optimality guarantees regardless of agent-environment boundaries and are applicable to various paradigms like state resetting and Monte-Carlo Tree Search. I have also revisited the assumptions underlying value-function approximation methods in batch RL, leading to new algorithms like BVFT that challenge existing hardness conjectures. In the realm of policy gradient methods, I introduced new variance reduction techniques using importance sampling estimators, resulting in improved efficiency and effectiveness. Additionally, I explored the intersection of symbolic regression and genetic programming, proposing methods like Control Variable Genetic Programming (CVGP) that outperform existing techniques in discovering symbolic expressions from data. My research also extends to off-policy evaluation in partially observable environments, where I developed new estimators that avoid exponential dependencies on the horizon, enhancing accuracy and generalizability. Overall, my work combines theoretical advancement with practical applicability, aiming to create algorithms that excel empirically while providing strong theoretical guarantees.
 
-  Here are the insights: Summary of Target Paper:\nThe target paper discusses the importance of learning low-dimensional vector representations of nodes in graphs, which has significantly advanced various tasks such as node classification and link prediction across multiple domains. It highlights successful applications of node embedding techniques in social networks, chemistry, and biology, emphasizing their relevance in enhancing graph-based tasks.
+  Here are the insights:
+  The developments in graph representation learning, particularly through context-aware approaches like CADE, could inspire further exploration of model-based reinforcement learning (RL) techniques. By integrating graph-based representations into RL frameworks, future research could focus on improving value-function approximation methods and off-policy evaluation in environments with complex relational structures. Additionally, adapting the local-to-global strategy for graph learning may provide insights into enhancing exploration strategies in RL, particularly in dynamic environments where agent interactions can be modeled as evolving graphs.
 
-  Keywords of Target Paper: Node Embedding, Graph Representation, Low-dimensional Vectors, Link Prediction, Node Classification
+  Here are the related works:
+  1th paper: This note clarifies some confusions (and perhaps throws out more) around model-based reinforcement learning and their theoretical understanding in the context of deep RL. Main topics of discussion are (1) how to reconcile model-based RL\'s bad empirical reputation on error compounding with its superior theoretical properties, and (2) the limitations of empirically popular losses. For the latter, concrete counterexamples for the "MuZero loss" are constructed to show that it not only fails in stochastic environments, but also suffers exponential sample complexity in deterministic environments when data provides sufficient coverage.
 
-  Valuable Points from Target Paper: The developments in graph representation learning, particularly through context-aware approaches like CADE, could inspire further exploration of model-based reinforcement learning (RL) techniques. By integrating graph-based representations into RL frameworks, future research could focus on improving value-function approximation methods and off-policy evaluation in environments with complex relational structures. Additionally, adapting the local-to-global strategy for graph learning may provide insights into enhancing exploration strategies in RL, particularly in dynamic environments where agent interactions can be modeled as evolving graphs.
-
-  Here are the related works: Paper: This note clarifies some confusions (and perhaps throws out more) around model-based reinforcement learning and their theoretical understanding in the context of deep RL. Main topics of discussion are (1) how to reconcile model-based RL\'s bad empirical reputation on error compounding with its superior theoretical properties, and (2) the limitations of empirically popular losses. For the latter, concrete counterexamples for the "MuZero loss" are constructed to show that it not only fails in stochastic environments, but also suffers exponential sample complexity in deterministic environments when data provides sufficient coverage.Paper: When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary, causing problems in theoretical analyses that provide optimality guarantees. We address this issue via a simple and novel boundary-invariant analysis of Fitted Q-Iteration, a representative RL algorithm, where the assumptions and the guarantees are invariant to the choice of boundary. We also discuss closely related issues on state resetting and Monte-Carlo Tree Search, deterministic vs stochastic systems, imitation learning, and the verifiability of theoretical assumptions from data.
+  2th paper: When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary, causing problems in theoretical analyses that provide optimality guarantees. We address this issue via a simple and novel boundary-invariant analysis of Fitted Q-Iteration, a representative RL algorithm, where the assumptions and the guarantees are invariant to the choice of boundary. We also discuss closely related issues on state resetting and Monte-Carlo Tree Search, deterministic vs stochastic systems, imitation learning, and the verifiability of theoretical assumptions from data.
 
   Please begin brainstorming idea conditioned on the "Keywords" and "Summary" of the target paper Please keep it within one to two sentences.
 - |
   Develop a graph-based reinforcement learning framework that leverages node embeddings to represent states and actions, enabling more efficient exploration and value function approximation in environments with complex relational structures, while incorporating boundary-invariant analyses to ensure optimality guarantees regardless of how the agent-environment boundary is defined.
 
 template: |
-  Here is your research background: {bio}
+  Here is your research background: 
+  {bio}
 
-  Here are the research insights: {insights}
+  Here are the research insights: 
+  {insights}
 
-  Here are the related works: {papers}
+  Here are the related works: 
+  {papers}
 
-  Please begin brainstorming idea conditioned on your research background. Please keep it within one to two sentences.
+  Please begin brainstorming idea conditioned on your research background. Please keep it within one to two sentences. Your idea should be different from those in the related papers.