分布式单词发音抓取机器人

　2012-05-24 09:22:06　来源：WEB开发网　　　

核心提示：说明：在构造函数中获取到为客户端文件回写线程服务的套接字，通过该套接字执行数据回写协议，分布式单词发音抓取机器人(3)，将数据写回到磁盘上，写磁盘线程的执行逻辑对应的流程图如图图5所示：图5 写磁盘线程任务请求线程class TaskRequestThread(threading.Thread): global

说明：在构造函数中获取到为客户端文件回写线程服务的套接字，通过该套接字执行数据回写协议，将数据写回到磁盘上。
写磁盘线程的执行逻辑对应的流程图如图图5所示：

图5 写磁盘线程
任务请求线程

class TaskRequestThread(threading.Thread): 
    global wordQueueMutex 
 
    def __init__(self,wordQueue): 
        threading.Thread.__init__(self) 
        self.requestSocket = socket(AF_INET,SOCK_STREAM) 
        self.taskQueue = wordQueue 
        self.requestSocket.connect(('localhost',wordSocketPort)) 
 
    def run(self): 
        while True: 
            wordQueueMutex.acquire() 
            if(len(self.taskQueue) < downloadWokerNum ): 
                requestStr = 'please' 
                self.requestSocket.send(requestStr.encode('ascii')) 
                rawStr = self.requestSocket.recv(rawStrLen)     #group 10 words once 
                wordsStr = rawStr.decode('ascii') 
                wordsList = wordsStr.split() 
                for word in wordsList: 
                    self.taskQueue.append(word) 
            exit = True 
            for word in self.taskQueue: 
                if word != '0': 
                    exit = False 
            if(exit == True): 
                self.requestSocket.send('0') 
                self.requestSocket.close() 
                wordQueueMutex.release() 
                break 
            wordQueueMutex.release() 
            time.sleep(taskRequestWorkerSleep)

说明：请求线程定期扫描任务队列中任务的数量，如果小于一定的阈值，则从主节点上请求新的单词列表任务。否则，线程进入睡眠。如果当前队列中的所有单词都为‘0’，说明主节点上已经没有单词任务了，这时可以退出了。
任务请求线程对应的执行逻辑如图6所示：

图6 任务请求线程
下载线程

class DownloadThread(threading.Thread): 
    global wordQueueMutex 
    global fileQueueMutex 
 
    def __init__(self,wordQueue,fileQueue): 
        threading.Thread.__init__(self) 
        self.wordQueue = wordQueue 
        self.fileQueue = fileQueue 
     
    def run(self): 
        while True: 
            time.sleep(2)           #delete this when presenting a demostration 
            word = '' 
            wordQueueMutex.acquire()             
            while (len(self.wordQueue) == 0): 
                wordQueueMutex.release() 
                time.sleep(downloadWorkerSleep) 
                wordQueueMutex.acquire() 
            if(self.wordQueue[0]=='0'): 
                wordQueueMutex.release() 
                fileQueueMutex.acquire() 
                self.fileQueue.append('0') 
                fileQueueMutex.release() 
                break; 
            else: 
                word = self.wordQueue.pop(0) 
                wordQueueMutex.release() 
                url = "http://www.dwds.de/?qu="+word 
                urlContent = urllib2.urlopen(url).read() 
                #print urlContent 
                urlList = re.findall('filename=http://media.dwds.de/dwds/media/sound/dwdswb_aussprache/.*\.mp3', urlContent) 
                try: 
                    finalUrl = urlList[0][9:] 
                    #print finalUrl 
                    soundData = urllib2.urlopen(finalUrl).read()  
                                saveName=word+'.mp3'+'.local'  
                    #print saveName 
                                outfd = open(saveName,'wb')  
                                outfd.write(soundData)  
                                outfd.close() 
                    fileQueueMutex.acquire() 
                    self.fileQueue.append(word)                  
                    fileQueueMutex.release() 
                    print '%s:              OK' % word 
                except: 
                    print '%s:              FAILED' % word 
                                finally: 
                    pass