本研究深入探讨了如何利用Winograd-schema挑战来评估大型语言模型（LLMs）的常识推理能力。具体而言，我们在流行的WinoGrande基准数据集上评估了不同规模的生成式模型。我们发布了WinoWhat，这是一个新的语料库，其中WinoGrande验证集的每个实例都进行了释义。此外，我们对五个常识知识类别中的挑战性能进行了评估，提供了更精细的见解，揭示了哪些类型的知识对LLMs更具挑战性。令人惊讶的是，所有模型在WinoWhat上的表现明显更差，这表明LLMs在WinoGrande上的推理能力可能被高估。为了验证这是否是由于基准数据集的记忆效应，我们将基准实例与LLM训练数据进行匹配，并构建了两个测试套件。观察结果显示，记忆效应对WinoGrande上的模型性能影响较小。

In this study, we take a closer look at how Winograd schema challenges can be used to evaluate common sense reasoning in LLMs. Specifically, we evaluate generative models of different sizes on the popular WinoGrande benchmark. We release WinoWhat, a new corpus, in which each instance of the WinoGrande validation set is paraphrased. Additionally, we evaluate the performance on the challenge across five common sense knowledge categories, giving more fine-grained insights on what types of knowledge are more challenging for LLMs. Surprisingly, all models perform significantly worse on WinoWhat, implying that LLM reasoning capabilities are overestimated on WinoGrande. To verify whether this is an effect of benchmark memorization, we match benchmark instances to LLM trainingdata and create two test-suites. We observe that memorization has a minimal effect on model performance on WinoGrande.