使用Python进行医学信息抽取

Question

我是一名护士，我知道python，但我不是专家，我只是用它来处理DNA序列。

我们有用人类语言编写的医院记录，我本应将这些数据插入到数据库或csv文件中，但它们超过5000行，这可能非常困难。所有数据都是以一致的格式写入的，让我给您展示一个示例

11/11/2010 - 09:00am : He got nausea, vomiting and died 4 hours later

我应该获得以下数据

Sex: Male
Symptoms: Nausea
    Vomiting
Death: True
Death Time: 11/11/2010 - 01:00pm

另一个例子

11/11/2010 - 09:00am : She got heart burn, vomiting of blood and died 1 hours later in the operation room

我得到了

Sex: Female
Symptoms: Heart burn
    Vomiting of blood
Death: True
Death Time: 11/11/2010 - 10:00am

当我说……时，顺序不一致。所以in是一个关键字，后面的所有文本都是一个位置，直到我找到另一个关键字

在一开始，他或她决定性别，得到......接下来是一组症状，我应该根据分隔符进行划分，分隔符可以是逗号、下划符或其他任何东西，但对于同一行是一致的

死了……几小时后还应该得到多少小时，有时病人还活着并出院....etc

这就是说，我们有很多约定，我认为如果我能用关键字和模式标记文本，我就能完成这项工作。所以如果你知道一个有用的函数/模块/教程/工具来做这件事，最好是用python (如果不是python，那么gui工具会更好)。

以下是一些信息：

there are a lot of rules to express various medical data but here are few examples
- Start with the same date/time format followed by a space followd by a colon followed by a space followed by He/She followed space followed by rules separated by and
- Rules:
    * got <symptoms>,<symptoms>,....
    * investigations were done <investigation>,<investigation>,<investigation>,......
    * received <drug or procedure>,<drug or procedure>,.....
    * discharged <digit> (hour|hours) later
    * kept under observation
    * died <digit> (hour|hours) later
    * died <digit> (hour|hours) later in <place>
other rules do exist but they follow the same idea

Answer 1

它使用dateutil解析日期(例如'11/11/2010 - 09:00am')，使用parsedatetime解析相对时间(例如'4小时后‘)：

import dateutil.parser as dparser
import parsedatetime.parsedatetime as pdt
import parsedatetime.parsedatetime_consts as pdc
import time
import datetime
import re
import pprint
pdt_parser = pdt.Calendar(pdc.Constants())   
record_time_pat=re.compile(r'^(.+)\s+:')
sex_pat=re.compile(r'\b(he|she)\b',re.IGNORECASE)
death_time_pat=re.compile(r'died\s+(.+hours later).*$',re.IGNORECASE)
symptom_pat=re.compile(r'[,-]')

def parse_record(astr):    
    match=record_time_pat.match(astr)
    if match:
        record_time=dparser.parse(match.group(1))
        astr,_=record_time_pat.subn('',astr,1)
    else: sys.exit('Can not find record time')
    match=sex_pat.search(astr)    
    if match:
        sex=match.group(1)
        sex='Female' if sex.lower().startswith('s') else 'Male'
        astr,_=sex_pat.subn('',astr,1)
    else: sys.exit('Can not find sex')
    match=death_time_pat.search(astr)
    if match:
        death_time,date_type=pdt_parser.parse(match.group(1),record_time)
        if date_type==2:
            death_time=datetime.datetime.fromtimestamp(
                time.mktime(death_time))
        astr,_=death_time_pat.subn('',astr,1)
        is_dead=True
    else:
        death_time=None
        is_dead=False
    astr=astr.replace('and','')    
    symptoms=[s.strip() for s in symptom_pat.split(astr)]
    return {'Record Time': record_time,
            'Sex': sex,
            'Death Time':death_time,
            'Symptoms': symptoms,
            'Death':is_dead}


if __name__=='__main__':
    tests=[('11/11/2010 - 09:00am : He got nausea, vomiting and died 4 hours later',
            {'Sex':'Male',
             'Symptoms':['got nausea', 'vomiting'],
             'Death':True,
             'Death Time':datetime.datetime(2010, 11, 11, 13, 0),
             'Record Time':datetime.datetime(2010, 11, 11, 9, 0)}),
           ('11/11/2010 - 09:00am : She got heart burn, vomiting of blood and died 1 hours later in the operation room',
           {'Sex':'Female',
             'Symptoms':['got heart burn', 'vomiting of blood'],
             'Death':True,
             'Death Time':datetime.datetime(2010, 11, 11, 10, 0),
             'Record Time':datetime.datetime(2010, 11, 11, 9, 0)})
           ]

    for record,answer in tests:
        result=parse_record(record)
        pprint.pprint(result)
        assert result==answer
        print

收益率：

{'Death': True,
 'Death Time': datetime.datetime(2010, 11, 11, 13, 0),
 'Record Time': datetime.datetime(2010, 11, 11, 9, 0),
 'Sex': 'Male',
 'Symptoms': ['got nausea', 'vomiting']}

{'Death': True,
 'Death Time': datetime.datetime(2010, 11, 11, 10, 0),
 'Record Time': datetime.datetime(2010, 11, 11, 9, 0),
 'Sex': 'Female',
 'Symptoms': ['got heart burn', 'vomiting of blood']}

注意:要小心解析日期。'8/9/2010‘是指8月9日，还是9月8日？是否所有的记录保持者都使用相同的约定？如果你选择使用dateutil (如果日期字符串没有严格的结构，我真的认为这是最好的选择)，一定要阅读dateutil documentation中关于“格式优先级”的部分，这样你就可以(希望)正确地解析'8/9/2010‘。如果您不能保证所有的记录保管器都使用相同的约定来指定日期，那么将手动检查此脚本的结果。这在任何情况下都可能是明智的。

Answer 2

以下是你可以解决这个问题的一些可能的方法-

1. Using正则表达式-根据文本中的模式定义它们。匹配表达式，提取模式，然后对所有记录重复。这种方法需要很好地理解数据的格式&当然是正则表达式、:)
2. String操作、-这种方法相对简单。同样，人们需要很好地理解数据的格式。这是我在下面所做的。
3. Machine -你可以定义你所有的规则，并根据这些规则训练一个模型。在此之后，模型尝试使用您提供的规则提取数据。这是一种比前两种方法更通用的方法。也是最难实现的。

看看这对你是否有效。可能需要一些调整。

new_file = open('parsed_file', 'w')
for rec in open("your_csv_file"):
    tmp = rec.split(' : ')
    date = tmp[0]
    reason = tmp[1]

    if reason[:2] == 'He':
        sex = 'Male'
        symptoms = reason.split(' and ')[0].split('He got ')[1]
    else:
        sex = 'Female'
        symptoms = reason.split(' and ')[0].split('She got ')[1]
    symptoms = [i.strip() for i in symptoms.split(',')]
    symptoms = '\n'.join(symptoms)
    if 'died' in rec:
        died = 'True'
    else:
        died = 'False'
    new_file.write("Sex: %s\nSymptoms: %s\nDeath: %s\nDeath Time: %s\n\n" % (sex, symptoms, died, date))

\n\n记录是换行符分隔的\n &因为您没有提到一个患者记录是两个换行符分隔的Ech。

护士后来：@你最终做了什么？只是好奇而已。

Answer 3

也许这也能帮到你，它还没有经过测试

import collections
import datetime
import re

retrieved_data = []

Data = collections.namedtuple('Patient', 'Sex, Symptoms, Death, Death_Time')
dict_data = {'Death':'',
             'Death_Time':'',
             'Sex' :'',
             'Symptoms':''}


with open('data.txt') as f:
     for line in iter(f.readline, ""):

         date, text = line.split(" : ")
         if 'died' in text:
             dict_data['Death'] = True
             dict_data['Death_Time'] = datetime.datetime.strptime(date, 
                                                                 '%d/%m/%Y - %I:%M%p')
             hours = re.findall('[\d]+', datetime.text)
             if hours:
                 dict_data['Death_Time'] += datetime.timedelta(hours=int(hours[0]))
         if 'she' in text:
            dict_data['Sex'] = 'Female'
         else:
            dict_data['Sex'] = 'Male'

         symptoms = text[text.index('got'):text.index('and')].split(',')

         dict_data['Symptoms'] = '\n'.join(symptoms) 

         retrieved_data.append(Data(**dict_data))

         # EDIT : Reset the data dictionary.
         dict_data = {'Death':'',
             'Death_Time':'',
             'Sex' :'',
             'Symptoms':''}