R字符串编码最佳实践

Question

我遇到了与本文中描述的相同的问题：identical strings from different data files won't match in R

这意味着，我有带有字符串的向量，这看起来完全相同，但是当set等于==时，返回一个false。

这里有一个例子：

> allowed_stock_exchanges
 [1] "Australian Securities Exchange"  "Borsa Italiana SpA"             
 [3] "Canadian Securities Exchange"    "Euronext Amsterdam"             
 [5] "Euronext Brussels"               "Euronext Lisbon"                
 [7] "Euronext Paris"                  "Frankfurt"                      
 [9] "Irish Stock Exchange"            "London Stock Exchange"          
[11] "Mercado Continuo Espanol (SIBE)" "NASDAQ"                         
[13] "NASDAQ OMX Stockholm"            "NYSE"                           
[15] "NYSE MKT LLC"                    "OMX Nordic Copenhagen"          
[17] "OMX Nordic Helsinki"             "Oslo Bors"                      
[19] "OTC"                             "Swiss SIX Exchange"             
[21] "Toronto"                         "Vienna Stock Exchange"          
[23] "XETRA"                          
> available_stock_exchanges
 [1] "NYSE"                            "NASDAQ"                         
 [3] "OTC"                             "NYSE MKT LLC"                   
 [5] "London Stock Exchange"           "TSX Venture Exchange"           
 [7] "Philippine Stock Exchange"       "Toronto"                        
 [9] "Australian Securities Exchange"  "Korea Stock Exchange"           
[11] "Kuala Lumpur"                    "New Zealand Exchange Ltd"       
[13] "Singapore"                       "XETRA"                          
[15] "Vienna Stock Exchange"           "Canadian Securities Exchange"   
[17] "Frankfurt"                       "NSX Australia"                  
[19] "NASDAQ OMX Stockholm"            "Mercado Continuo Espanol (SIBE)"
[21] "Euronext Paris"                  "Euronext Brussels"              
[23] "OMX Nordic Copenhagen"           "Swiss SIX Exchange"             
[25] "Euronext Amsterdam"              "Borsa Italiana SpA"             
[27] "OMX Nordic Helsinki"             "Oslo Bors"                      
[29] "Euronext Lisbon"                 "Dusseldorf"                     
[31] "Irish Stock Exchange"            "Hamburg Stock Exchange"         
[33] "Luxembourg"                      "OMX Nordic Iceland"             
[35] "Warsaw Stock Exchange"           "Norwegian OTC Market"           
[37] "Buenos Aires"                    "Berlin"                         
[39] "Hong Kong"                       "Berne Stock Exchange"           
[41] "Johannesburg"                    "Nordic Growth Market"           
[43] "Athens Stock Exchange"          
> allowed_stock_exchanges[1] == available_stock_exchanges[9]
[1] FALSE

> charToRaw(allowed_stock_exchanges[1])
 [1] 41 75 73 74 72 61 6c 69 61 6e c2 a0 53 65 63 75 72 69 74 69 65 73 c2 a0 45 78 63 68 61 6e 67
[32] 65
> charToRaw(available_stock_exchanges[9])
 [1] 41 75 73 74 72 61 6c 69 61 6e 20 53 65 63 75 72 69 74 69 65 73 20 45 78 63 68 61 6e 67 65
> Encoding(allowed_stock_exchanges)
 [1] "UTF-8"   "UTF-8"   "UTF-8"   "UTF-8"   "UTF-8"   "UTF-8"   "UTF-8"   "unknown" "UTF-8"  
[10] "UTF-8"   "UTF-8"   "unknown" "UTF-8"   "unknown" "UTF-8"   "UTF-8"   "UTF-8"   "UTF-8"  
[19] "unknown" "UTF-8"   "unknown" "UTF-8"   "unknown"
> Encoding(available_stock_exchanges)
 [1] "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown"
[10] "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown"
[19] "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown"
[28] "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown"
[37] "unknown" "unknown" "unknown" "unknown" "unknown" "unknown" "unknown"

我在混乱中的问题：

1. 避免此类问题的最佳实践是什么？例如，当从包含字符串的不同来源读取数据时，应该检查/调整什么？

1. ，我现在如何使这两个向量均一，这样如果设置为相等，它将真正地向我表明它们是相等的？

有什么建议吗？

编辑:我发现未知数是ASCII。然而，不幸的是，我不知道如何“同质化”一切

Answer 1

charToRaw()调用信息非常丰富。一个字符串将空格编码为原始字符20 ( ASCII空间)，另一个字符串将其编码为C2 A0，即“无中断空间”。R认为这些是不同的，所以字符串不匹配。

要解决这个问题，您可能需要编写一个小函数，将所有内容转换为一致的格式。例如，要处理不可中断的空间问题，可以使用

allowed_stock_exchanges <- gsub("\u00a0", " ", allowed_stock_exchanges)

其中，\u00a0是该字符的Unicode编码；在UTF-8中，它显示为C2 A0。(有关这些关系，请参见https://www.utf8-chartable.de/这样的网站。)

使用tools::showNonASCII()函数可以帮助检测其他潜在问题。它会识别出像不破的空间之类的东西。它不会将自己限制在字符串看起来相同但编码不同的情况下，所以它也会识别重音字符。

另一种可能是使用

allowed_stock_exchanges <- iconv(allowed_stock_exchanges, to = "ASCII//TRANSLIT")

它试图转换为ASCII，在必要时进行替换。这将将重音字符转换为一些有趣的ASCII版本(我看到"é“转换为”e“)，因此在比较之前，您需要在两个字符串上使用它。但根据您的系统，转换可能与我的不一样。