当前位置:   article > 正文

django+django-haystack+Whoosh(后期切换引擎为Elasticsearch+ik)+Jieba+mysql_django +whoosh

django +whoosh

1.前提准备

环境介绍

  • haystack是django的开源搜索框架,该框架支持Solr, Elasticsearch, Whoosh, *Xapian*搜索引擎,不用更改代码,直接切换引擎,减少代码量。

  • 搜索引擎使用Whoosh,这是一个由纯Python实现的全文搜索引擎,没有二进制文件等,比较小巧,配置比较简单,当然性能自然略低。whoosh和xapian的性能差距还是比较明显。索引和搜索的速度有近4倍的差距,在full cache情况下的性能差距更是达到了60倍。

  • 中文分词+,由于Whoosh自带的是英文分词,对中文的分词支持不是太好,故用jieba替换whoosh的分词组件。

  • Elasticsearch:开源的搜索引擎,本文版本为7.6.0

  • 其他:Python3.6.5, Django2.2 

安装环境

  1. pip3 install django==2.2 -i https://pypi.douban.com/simple
  2. pip3 install whoosh -i https://pypi.douban.com/simple
  3. pip3 install django-haystack -i https://pypi.douban.com/simple
  4. pip3 install jieba -i https://pypi.douban.com/simple
  5. pip3 install pymysql -i https://pypi.douban.com/simple
  6. pip3 install elasticsearch==7.6.0 -i https://pypi.douban.com/simple/

项目结构

  1. - Project
  2. - Project
  3. - settings.py
  4. - blog
  5. - models.py

表结构

models.py

  1. from django.db import models
  2. class UserInfo(models.Model):
  3. username = models.CharField(verbose_name='用户名', max_length=225)
  4. def __str__(self):
  5. return self.username
  6. class Tag(models.Model):
  7. name = models.CharField(verbose_name='标签名称', max_length=225)
  8. def __str__(self):
  9. return self.name
  10. class Article(models.Model):
  11. title = models.CharField(verbose_name='标题', max_length=225)
  12. content = models.CharField(verbose_name='内容', max_length=225)
  13. # 外键
  14. username = models.ForeignKey(verbose_name='用户', to='UserInfo', on_delete=models.DO_NOTHING)
  15. tag = models.ForeignKey(verbose_name='标签', to='Tag', on_delete=models.DO_NOTHING)
  16. def __str__(self):
  17. return self.title

图解

本文优势

集全网的django+django-haystack+Whoosh的总结,取其精华,去其糟粕,加入了新的注解。

如果你想你的es或者Whoosh集成到django上,那你来对地方了

django+django-haystack+Whoosh+Jieba+mysql

1. setting.py配置

  1. # 数据库配置
  2. DATABASES = {
  3. 'default': {
  4. 'ENGINE': 'django.db.backends.mysql',
  5. 'NAME': 'dj_ha',
  6. 'USER': 'root',
  7. 'PASSWORD': 'foobared',
  8. 'HOST': '106.14.42.253',
  9. 'PORT': '11111',
  10. }
  11. }
  12. # app
  13. INSTALLED_APPS = [
  14. 'haystack',
  15. ]
  16. # 本教程使用的是Whoosh,故配置如下
  17. HAYSTACK_CONNECTIONS = {
  18. 'default': {
  19. 'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
  20. 'PATH': os.path.join(os.path.dirname(__file__), 'whoosh_index'),
  21. },
  22. }
  23. # 自动更新索引
  24. HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'
  25. # 设置每页显示的数目,默认为20,可以自己修改
  26. HAYSTACK_SEARCH_RESULTS_PER_PAGE = 8

2. 为表模型创建索引,search_indexes.py

1. 如果你想针对某个app,例如blog做全文检索,则必须在blog的目录下面,建立search_indexes.py文件,文件名不能修改,必须叫search_indexes.py

  1. from haystack import indexes
  2. from .models import Article
  3. # ArticleIndex:固定写法 表名Index
  4. class ArticleIndex(indexes.SearchIndex, indexes.Indexable):
  5. # 固定写法 document=True:haystack和搜索引擎,将给text字段分词,建立索引,使用此字段的内容作为索引进行检索
  6. # use_template=True,使用自己的模板,与document=True进行搭配,自定义检索字段模板(允许谁可以被全文检索,就是谁被建立索引)
  7. text = indexes.CharField(document=True, use_template=True)
  8. # 以下字段作为辅助数据,便于调用,最后也不知道怎么辅助,我注释了,也不影响搜索
  9. # title:写入引擎的字段名,model_attr='title':相对应的表模型字段名,
  10. title = indexes.CharField(model_attr='title')
  11. content = indexes.CharField(model_attr='content')
  12. username = indexes.CharField(model_attr='username')
  13. tag = indexes.CharField(model_attr='tag')
  14. def get_model(self):
  15. # 需要建立索引的模型
  16. return Article
  17. def index_queryset(self, using=None):
  18. """Used when the entire index for model is updated."""
  19. # 写入引擎的数据,必须返回queryset类型
  20. return self.get_model().objects.all()

3. 创建被检索的模板(允许谁可以全文检索)

这个数据模板的作用是对Article.title, Article.content,Article.username.username

这三个字段建立索引,当检索的时候会对这三个字段的内容,做全文检索匹配。

数据模板的路径为yourapp/templates/search/indexes/yourapp/note_text.txt,

例如本例子为blog/templates/search/indexes/blog/article_text.txt  文件名必须为要索引的小写模型类名_text.txt

  1. {{ object.title }}
  2. {{ object.content }}
  3. {{ object.username.username }}

4. 路由

urls.py配置(用内置的视图,后期可以自定义,本文也有介绍)

  1. # urls.py
  2. from django.contrib import admin
  3. from django.urls import path, include, re_path
  4. urlpatterns = [
  5. path('admin/', admin.site.urls),
  6. # 配置的搜索路由,路由可以自定义,include('haystack.urls')固定
  7. re_path(r'^search/', include('haystack.urls')),
  8. ]

 haystack.urls的内容(内置的,只是我拉出来,让你看一下,不需要进行修改)

  1. from django.urls import path
  2. from haystack.views import SearchView
  3. urlpatterns = [path("", SearchView(), name="haystack_search")]

5. search.html

SearchView()视图函数默认使用的html模板为当前app目录下,

路径为app名称,/templates/search/search.html

所以需要在blog/templates/search/下添加search.html文件,内容为

 search.html(原生)

  1. <h2>Search</h2>
  2. <style>
  3. span.highlighted {
  4. color: red;
  5. }
  6. </style>
  7. <!--高亮加载-->
  8. {% load highlight %}
  9. <form method="get" action=".">
  10. <table>
  11. <!-- 对象.as_table 生成表格,里边会自动成成input标签 -->
  12. {{ form.as_table }}
  13. {# {{ form.title.label }}#}
  14. <tr>
  15. <td></td>
  16. <td>
  17. <input type="submit" value="Search">
  18. </td>
  19. </tr>
  20. </table>
  21. {% if query %}
  22. <h3>返回结果</h3>
  23. {% for result in page.object_list %}
  24. <!-- page.object_list:返回查询的一页数据 -->
  25. <!-- result:数据对象 -->
  26. <p>
  27. {# <a href="{{ result.object.get_absolute_url }}">{{ result.object.title }}</a>#}
  28. <a href="{{ result.object.get_absolute_url }}">{% highlight result.object.title with query %}</a>
  29. </p>
  30. <span>
  31. {% highlight result.object.content with query %}
  32. {# {{ result.object.content }}#}
  33. </span>
  34. {% empty %}
  35. <p>没有查询到结果!!!</p>
  36. {% endfor %}
  37. <!-- 分页 -->
  38. {% if page.has_previous or page.has_next %}
  39. <div>
  40. {% if page.has_previous %}<a href="?q={{ query }}&page={{ page.previous_page_number }}">{% endif %}«
  41. Previous{% if page.has_previous %}</a>{% endif %}
  42. |
  43. {% if page.has_next %}<a href="?q={{ query }}&page={{ page.next_page_number }}">{% endif %}Next »
  44. {% if page.has_next %}</a>{% endif %}
  45. </div>
  46. {% endif %}
  47. {% else %}
  48. {# Show some example queries to run, maybe query syntax, something else? #}
  49. {% endif %}
  50. </form>

 后端返回数据介绍

  1. # print(context)
  2. """
  3. {
  4. 'query': '刘',
  5. 'form': <ModelSearchForm bound=True, valid=True, fields=(q;models)>,
  6. 'page': <Page 1 of 1>,
  7. 'paginator': <django.core.paginator.Paginator object at 0x0000017D7E0F3470>,
  8. 'suggestion': None}
  9. """
  10. # print(context.get('page').__dict__)
  11. """
  12. {
  13. 'object_list':
  14. [
  15. <SearchResult: blog.article (pk=6)>,
  16. <SearchResult: blog.article (pk=8)>,
  17. <SearchResult: blog.article (pk=1)>
  18. ],
  19. 'number': 1,
  20. 'paginator': <django.core.paginator.Paginator object at 0x00000257C11A65C0>
  21. }
  22. """

前端返回数据介绍

  1. {% load highlight %}:高亮加载 内置的会省略搜到的内容,之前的内容
  2. {% load my_filters_and_tags %}:自定义高亮
  3. form.as_table:生成表格,里边会自动成成input标签
  4. query:查询的参数
  5. page.object_list:返回的查询一页数据
  6. result:数据对象集
  7. result.object:当前查询的数据对象
  8. page.has_previous or page.has_next:分页

6. 高亮配置 

  1. # 7.高亮加载
  2. <style>
  3. span.highlighted {
  4. color: red;
  5. }
  6. </style>
  7. # 1.使用默认值
  8. {% highlight result.summary with query %}
  9. # 案例
  10. <a href="{{ result.object.get_absolute_url }}">
  11. {% highlight result.object.title with query %}
  12. </a>
  13. # 2.这里我们为 {{ result.summary }}里所有的 {{ query }} 指定了一个<div></div>标签,并且将class设置为highlight_me_please,这样就可以自己通过CSS为{{ query }}添加高亮效果了,怎么样,是不是很科学呢
  14. {% highlight result.summary with query html_tag "div" css_class "highlight_me_please" %}
  15. # 3.这里可以限制最终{{ result.summary }}被高亮处理后的长度
  16. {% highlight result.summary with query max_length 40 %}
  17. # 5.自定义使用(后面会介绍)
  18. # 5.4格式
  19. {% myhighlight <text_block> with <query> [css_class "class_name"] [html_tag "span"] [max_length 200] [start_head True] %}
  20. # 5.2使用一
  21. {% myhighlight result.object.content with query css_class "highlighted" html_tag "span" max_length 200 start_head True %}
  22. # 5.3自定义二
  23. {% myhighlight result.object.content with query css_class "highlighted" start_head True %}

7.自定义

自定义返回内容

在app下新建一个文件名称search_views

  1. # 重写SearchView,实现自定义内容
  2. # blog/search_views.py
  3. from haystack.views import SearchView
  4. # 导入模块
  5. from .models import *
  6. class MySeachView(SearchView):
  7. def extra_context(self): # 重载extra_context来添加额外的context内容
  8. context = super(MySeachView, self).extra_context()
  9. my_str = '111'
  10. context['my_str'] = my_str
  11. # print(context)
  12. return context

修改路由

  1. from django.contrib import admin
  2. from django.urls import path, include, re_path
  3. from blog import search_views
  4. urlpatterns = [
  5. path('admin/', admin.site.urls),
  6. # 原生的
  7. # re_path(r'^search/', include('haystack.urls')),
  8. # 自己的
  9. re_path(r'^search/', search_views.MySeachView(), name='haystack_search'),
  10. ]

前端使用 

  1. <div>
  2. 圆明园:{{ my_str }}
  3. </div>

自定义search.html模板 

1. 保证有一个from,get请求,input标签的name=q,value=Search,

  1. <form method="get" action=".">
  2. <table>
  3. <tr>
  4. <th>
  5. <label for="id_q">Search:</label>
  6. </th>
  7. <td>
  8. <input type="search" name="q" value="不得不说" id="id_q">
  9. </td>
  10. </tr>
  11. <tr>
  12. <td>
  13. <input type="submit" value="Search">
  14. </td>
  15. </tr>
  16. </table>
  17. </form>

自定义高亮显示(原生的会省略)

新建文件夹templatetags

添加blog/templatetags/my_filters_and_tags.py 文件和 blog/templatetags/highlighting.py 文件,

内容如下(源码分别位于haystack/templatetags/lighlight.py 和 haystack/utils/lighlighting.py 中):
my_filters_and_tags.py

  1. # encoding: utf-8
  2. from __future__ import absolute_import, division, print_function, unicode_literals
  3. from django import template
  4. from django.conf import settings
  5. from django.core.exceptions import ImproperlyConfigured
  6. from django.utils import six
  7. from haystack.utils import importlib
  8. register = template.Library()
  9. class HighlightNode(template.Node):
  10. def __init__(self, text_block, query, html_tag=None, css_class=None, max_length=None, start_head=None):
  11. self.text_block = template.Variable(text_block)
  12. self.query = template.Variable(query)
  13. self.html_tag = html_tag
  14. self.css_class = css_class
  15. self.max_length = max_length
  16. self.start_head = start_head
  17. if html_tag is not None:
  18. self.html_tag = template.Variable(html_tag)
  19. if css_class is not None:
  20. self.css_class = template.Variable(css_class)
  21. if max_length is not None:
  22. self.max_length = template.Variable(max_length)
  23. if start_head is not None:
  24. self.start_head = template.Variable(start_head)
  25. def render(self, context):
  26. text_block = self.text_block.resolve(context)
  27. query = self.query.resolve(context)
  28. kwargs = {}
  29. if self.html_tag is not None:
  30. kwargs['html_tag'] = self.html_tag.resolve(context)
  31. if self.css_class is not None:
  32. kwargs['css_class'] = self.css_class.resolve(context)
  33. if self.max_length is not None:
  34. kwargs['max_length'] = self.max_length.resolve(context)
  35. if self.start_head is not None:
  36. kwargs['start_head'] = self.start_head.resolve(context)
  37. # Handle a user-defined highlighting function.
  38. if hasattr(settings, 'HAYSTACK_CUSTOM_HIGHLIGHTER') and settings.HAYSTACK_CUSTOM_HIGHLIGHTER:
  39. # Do the import dance.
  40. try:
  41. path_bits = settings.HAYSTACK_CUSTOM_HIGHLIGHTER.split('.')
  42. highlighter_path, highlighter_classname = '.'.join(path_bits[:-1]), path_bits[-1]
  43. highlighter_module = importlib.import_module(highlighter_path)
  44. highlighter_class = getattr(highlighter_module, highlighter_classname)
  45. except (ImportError, AttributeError) as e:
  46. raise ImproperlyConfigured("The highlighter '%s' could not be imported: %s" % (settings.HAYSTACK_CUSTOM_HIGHLIGHTER, e))
  47. else:
  48. from .highlighting import Highlighter
  49. highlighter_class = Highlighter
  50. highlighter = highlighter_class(query, **kwargs)
  51. highlighted_text = highlighter.highlight(text_block)
  52. return highlighted_text
  53. @register.tag
  54. def myhighlight(parser, token):
  55. """
  56. Takes a block of text and highlights words from a provided query within that
  57. block of text. Optionally accepts arguments to provide the HTML tag to wrap
  58. highlighted word in, a CSS class to use with the tag and a maximum length of
  59. the blurb in characters.
  60. Syntax::
  61. {% highlight <text_block> with <query> [css_class "class_name"] [html_tag "span"] [max_length 200] %}
  62. Example::
  63. # Highlight summary with default behavior.
  64. {% highlight result.summary with request.query %}
  65. # Highlight summary but wrap highlighted words with a div and the
  66. # following CSS class.
  67. {% highlight result.summary with request.query html_tag "div" css_class "highlight_me_please" %}
  68. # Highlight summary but only show 40 characters.
  69. {% highlight result.summary with request.query max_length 40 %}
  70. """
  71. bits = token.split_contents()
  72. tag_name = bits[0]
  73. if not len(bits) % 2 == 0:
  74. raise template.TemplateSyntaxError(u"'%s' tag requires valid pairings arguments." % tag_name)
  75. text_block = bits[1]
  76. if len(bits) < 4:
  77. raise template.TemplateSyntaxError(u"'%s' tag requires an object and a query provided by 'with'." % tag_name)
  78. if bits[2] != 'with':
  79. raise template.TemplateSyntaxError(u"'%s' tag's second argument should be 'with'." % tag_name)
  80. query = bits[3]
  81. arg_bits = iter(bits[4:])
  82. kwargs = {}
  83. for bit in arg_bits:
  84. if bit == 'css_class':
  85. kwargs['css_class'] = six.next(arg_bits)
  86. if bit == 'html_tag':
  87. kwargs['html_tag'] = six.next(arg_bits)
  88. if bit == 'max_length':
  89. kwargs['max_length'] = six.next(arg_bits)
  90. if bit == 'start_head':
  91. kwargs['start_head'] = six.next(arg_bits)
  92. return HighlightNode(text_block, query, **kwargs)

highlighting.py

  1. # encoding: utf-8
  2. from __future__ import absolute_import, division, print_function, unicode_literals
  3. from django.utils.html import strip_tags
  4. class Highlighter(object):
  5. #默认值
  6. css_class = 'highlighted'
  7. html_tag = 'span'
  8. max_length = 200
  9. start_head = False
  10. text_block = ''
  11. def __init__(self, query, **kwargs):
  12. self.query = query
  13. if 'max_length' in kwargs:
  14. self.max_length = int(kwargs['max_length'])
  15. if 'html_tag' in kwargs:
  16. self.html_tag = kwargs['html_tag']
  17. if 'css_class' in kwargs:
  18. self.css_class = kwargs['css_class']
  19. if 'start_head' in kwargs:
  20. self.start_head = kwargs['start_head']
  21. self.query_words = set([word.lower() for word in self.query.split() if not word.startswith('-')])
  22. def highlight(self, text_block):
  23. self.text_block = strip_tags(text_block)
  24. highlight_locations = self.find_highlightable_words()
  25. start_offset, end_offset = self.find_window(highlight_locations)
  26. return self.render_html(highlight_locations, start_offset, end_offset)
  27. def find_highlightable_words(self):
  28. # Use a set so we only do this once per unique word.
  29. word_positions = {}
  30. # Pre-compute the length.
  31. end_offset = len(self.text_block)
  32. lower_text_block = self.text_block.lower()
  33. for word in self.query_words:
  34. if not word in word_positions:
  35. word_positions[word] = []
  36. start_offset = 0
  37. while start_offset < end_offset:
  38. next_offset = lower_text_block.find(word, start_offset, end_offset)
  39. # If we get a -1 out of find, it wasn't found. Bomb out and
  40. # start the next word.
  41. if next_offset == -1:
  42. break
  43. word_positions[word].append(next_offset)
  44. start_offset = next_offset + len(word)
  45. return word_positions
  46. def find_window(self, highlight_locations):
  47. best_start = 0
  48. best_end = self.max_length
  49. # First, make sure we have words.
  50. if not len(highlight_locations):
  51. return (best_start, best_end)
  52. words_found = []
  53. # Next, make sure we found any words at all.
  54. for word, offset_list in highlight_locations.items():
  55. if len(offset_list):
  56. # Add all of the locations to the list.
  57. words_found.extend(offset_list)
  58. if not len(words_found):
  59. return (best_start, best_end)
  60. if len(words_found) == 1:
  61. return (words_found[0], words_found[0] + self.max_length)
  62. # Sort the list so it's in ascending order.
  63. words_found = sorted(words_found)
  64. # We now have a denormalized list of all positions were a word was
  65. # found. We'll iterate through and find the densest window we can by
  66. # counting the number of found offsets (-1 to fit in the window).
  67. highest_density = 0
  68. if words_found[:-1][0] > self.max_length:
  69. best_start = words_found[:-1][0]
  70. best_end = best_start + self.max_length
  71. for count, start in enumerate(words_found[:-1]):
  72. current_density = 1
  73. for end in words_found[count + 1:]:
  74. if end - start < self.max_length:
  75. current_density += 1
  76. else:
  77. current_density = 0
  78. # Only replace if we have a bigger (not equal density) so we
  79. # give deference to windows earlier in the document.
  80. if current_density > highest_density:
  81. best_start = start
  82. best_end = start + self.max_length
  83. highest_density = current_density
  84. return (best_start, best_end)
  85. def render_html(self, highlight_locations=None, start_offset=None, end_offset=None):
  86. # Start by chopping the block down to the proper window.
  87. #text_block为内容,start_offset,end_offset分别为第一个匹配query开始和按长度截断位置
  88. text = self.text_block[start_offset:end_offset]
  89. # Invert highlight_locations to a location -> term list
  90. term_list = []
  91. for term, locations in highlight_locations.items():
  92. term_list += [(loc - start_offset, term) for loc in locations]
  93. loc_to_term = sorted(term_list)
  94. # Prepare the highlight template
  95. if self.css_class:
  96. hl_start = '<%s class="%s">' % (self.html_tag, self.css_class)
  97. else:
  98. hl_start = '<%s>' % (self.html_tag)
  99. hl_end = '</%s>' % self.html_tag
  100. # Copy the part from the start of the string to the first match,
  101. # and there replace the match with a highlighted version.
  102. #matched_so_far最终求得为text中最后一个匹配query的结尾
  103. highlighted_chunk = ""
  104. matched_so_far = 0
  105. prev = 0
  106. prev_str = ""
  107. for cur, cur_str in loc_to_term:
  108. # This can be in a different case than cur_str
  109. actual_term = text[cur:cur + len(cur_str)]
  110. # Handle incorrect highlight_locations by first checking for the term
  111. if actual_term.lower() == cur_str:
  112. if cur < prev + len(prev_str):
  113. continue
  114. #分别添上每个query+其后面的一部分(下一个query的前一个位置)
  115. highlighted_chunk += text[prev + len(prev_str):cur] + hl_start + actual_term + hl_end
  116. prev = cur
  117. prev_str = cur_str
  118. # Keep track of how far we've copied so far, for the last step
  119. matched_so_far = cur + len(actual_term)
  120. # Don't forget the chunk after the last term
  121. #加上最后一个匹配的query后面的部分
  122. highlighted_chunk += text[matched_so_far:]
  123. #如果不要开头not start_head才加点
  124. if start_offset > 0 and not self.start_head:
  125. highlighted_chunk = '...%s' % highlighted_chunk
  126. if end_offset < len(self.text_block):
  127. highlighted_chunk = '%s...' % highlighted_chunk
  128. #可见到目前为止还不包含start_offset前面的,即第一个匹配的前面的部分(text_block[:start_offset]),如需展示(当start_head为True时)便加上
  129. if self.start_head:
  130. highlighted_chunk = self.text_block[:start_offset] + highlighted_chunk
  131. return highlighted_chunk

前端使用

  1. <style>
  2. span.highlighted {
  3. color: red;
  4. }
  5. </style>
  6. {% load my_filters_and_tags %}
  7. {% myhighlight result.object.content with query css_class "highlighted" html_tag "span" max_length 200 start_head True %}

 8. 目前位置搜索已经完成,可以重建索引,同步数据,测试一下

python manage.py rebuild_index

9.jieba分词器配置

9.1 先从python包中复制whoosh_backend.py到app中,并改名为whoosh_cn_backend.py

文件路径:\site-packages\haystack\backends\whoosh_backend.py

 在这里插入图片描述

复制到的路径:

9.2 对whoosh_cn_backend.py做以下修改:

  1. 1、导入 ChineseAnalyze
  2. from jieba.analyse import ChineseAnalyzer
  3. 2、替换schema_fields[field_class.index_fieldname] = TEXT(下的analyzer
  4. analyzer=ChineseAnalyzer(),

 9.3 在django的配置文件中,修改搜索引擎

  1. HAYSTACK_CONNECTIONS = {
  2. 'default': {
  3. # 设置haystack的搜索引擎
  4. 'ENGINE': 'blog.whoosh_cn_backend.WhooshEngine',
  5. # 'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
  6. # 设置索引文件的位置
  7. 'PATH': os.path.join(BASE_DIR, 'whoosh_index'),
  8. }
  9. }

10 django+django-haystack+Elasticsearch7.5+ik+mysql

10.0 切换成es引擎,除了settings.py和把jieba换成ik,其他步骤跟上面的都一样

如果一开始,就是奔着es+ik来的,那步骤9 jieba分词器配置 不用看,直接从步骤8跳到这里来

10.1 安装es,ik

基于docker安装Elasticsearch+ElasticSearch-Head+IK分词器_骑台风走的博客-CSDN博客基于docker安装Elasticsearch+ElasticSearch-Head+IK分词器https://blog.csdn.net/qq_52385631/article/details/126567059?spm=1001.2014.3001.5501

10.2 使用ik重写es7.5引擎

10.2.1 新建elasticsearch_ik_backend.py(在自己的app下)

在 blog应用下新建名为 elasticsearch7_ik_backend.py 的文件,继承 Elasticsearch7SearchBackend(后端) 和 Elasticsearch7SearchEngine(搜索引擎) 并重写建立索引时的分词器设置

  1. from haystack.backends.elasticsearch7_backend import Elasticsearch7SearchBackend, Elasticsearch7SearchEngine
  2. """
  3. 分析器主要有两种情况会被使用:
  4. 第一种是插入文档时,将text类型的字段做分词然后插入倒排索引,
  5. 第二种就是在查询时,先对要查询的text类型的输入做分词,再去倒排索引搜索
  6. 如果想要让 索引 和 查询 时使用不同的分词器,ElasticSearch也是能支持的,只需要在字段上加上search_analyzer参数
  7. 在索引时,只会去看字段有没有定义analyzer,有定义的话就用定义的,没定义就用ES预设的
  8. 在查询时,会先去看字段有没有定义search_analyzer,如果没有定义,就去看有没有analyzer,再没有定义,才会去使用ES预设的
  9. """
  10. DEFAULT_FIELD_MAPPING = {
  11. "type": "text",
  12. "analyzer": "ik_max_word",
  13. # "analyzer": "ik_smart",
  14. "search_analyzer": "ik_smart"
  15. }
  16. class Elasticsearc7IkSearchBackend(Elasticsearch7SearchBackend):
  17. def __init__(self, *args, **kwargs):
  18. self.DEFAULT_SETTINGS['settings']['analysis']['analyzer']['ik_analyzer'] = {
  19. "type": "custom",
  20. "tokenizer": "ik_max_word",
  21. # "tokenizer": "ik_smart",
  22. }
  23. super(Elasticsearc7IkSearchBackend, self).__init__(*args, **kwargs)
  24. class Elasticsearch7IkSearchEngine(Elasticsearch7SearchEngine):
  25. backend = Elasticsearc7IkSearchBackend

 10.3 修改settings.py(切换成功)

  1. # es 7.x配置
  2. HAYSTACK_CONNECTIONS = {
  3. 'default': {
  4. # 'ENGINE': 'haystack.backends.elasticsearch7_backend.Elasticsearch7SearchEngine',
  5. 'ENGINE': 'blog.elasticsearch_ik_backend.Elasticsearch7IkSearchEngine',
  6. # 'URL': 'http://106.14.42.253:9200/',
  7. 'URL': 'http://106.14.42.253:9200/',
  8. # elasticsearch建立的索引库的名称,一般使用项目名作为索引库
  9. 'INDEX_NAME': 'elastic_new',
  10. },
  11. }

10.4 重建索引,同步数据

python manage.py rebuild_index

10.5 补充

10.5.1 未成功切换成ik

haystack 原先加载的是 ...\venv\Lib\site-packages\haystack\backends 文件夹下的 elasticsearch7_backend.py 文件,打开即可看到 elasticsearch7 引擎的默认配置 

若用上述方法建立出来的索引字段仍使用 snowball 分词器,则将原先elasticsearch7_backend.py 文件中的 DEFAULT_FIELD_MAPPING 也修改为 ik 分词器(或许是因为版本问题)

位置:D:\py_virtualenv\dj_ha\Lib\site-packages\haystack\backends\elasticsearch7_backend.py

修改内容:

  1. DEFAULT_FIELD_MAPPING = {
  2. "type": "text",
  3. "analyzer": "ik_max_word",
  4. "search_analyzer": "ik_smart",
  5. }

10.5.2 es6版本加入ik,重写引擎

  1. from haystack.backends.elasticsearch_backend import ElasticsearchSearchBackend
  2. from haystack.backends.elasticsearch_backend import ElasticsearchSearchEngine
  3. class IKSearchBackend(ElasticsearchSearchBackend):
  4. DEFAULT_ANALYZER = "ik_max_word" # 这里将 es 的 默认 analyzer 设置为 ik_max_word
  5. def __init__(self, connection_alias, **connection_options):
  6. super().__init__(connection_alias, **connection_options)
  7. def build_schema(self, fields):
  8. content_field_name, mapping = super(IKSearchBackend, self).build_schema(fields)
  9. for field_name, field_class in fields.items():
  10. field_mapping = mapping[field_class.index_fieldname]
  11. if field_mapping["type"] == "string" and field_class.indexed:
  12. if not hasattr(
  13. field_class, "facet_for"
  14. ) and not field_class.field_type in ("ngram", "edge_ngram"):
  15. field_mapping["analyzer"] = getattr(
  16. field_class, "analyzer", self.DEFAULT_ANALYZER
  17. )
  18. mapping.update({field_class.index_fieldname: field_mapping})
  19. return content_field_name, mapping
  20. class IKSearchEngine(ElasticsearchSearchEngine):
  21. backend = IKSearchBackend

11.实时更新索原理:采用信号

配置

  1. # 在django配置文件中,添加索引值,文章更新的时候,就会自动更新索引值
  2. HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'

RealtimeSignalProcessor源码如下:

  1. class RealtimeSignalProcessor(BaseSignalProcessor):
  2. """
  3. Allows for observing when saves/deletes fire & automatically updates the
  4. search engine appropriately.
  5. 当 检索对象出现保存或者删除的时候更新索引值。
  6. """
  7. def setup(self):
  8. # Naive (listen to all model saves).
  9. models.signals.post_save.connect(self.handle_save)
  10. models.signals.post_delete.connect(self.handle_delete)
  11. # Efficient would be going through all backends & collecting all models
  12. # being used, then hooking up signals only for those.
  13. def teardown(self):
  14. # Naive (listen to all model saves).
  15. models.signals.post_save.disconnect(self.handle_save)
  16. models.signals.post_delete.disconnect(self.handle_delete)
  17. # Efficient would be going through all backends & collecting all models
  18. # being used, then disconnecting signals only for those.

本文借鉴

Django haystack实现全文搜索 - -零 - 博客园 (cnblogs.com)
(9条消息) django-haystack全文检索详细教程_AC_hell的博客-CSDN博客
(9条消息) Django全文检索Haystack模块_NQ31的博客-CSDN博客_django haystack
(9条消息) django+drf_haystack+elasticsearch_骑台风走的博客-CSDN博客

(5条消息) Haystack 使用 Elasticsearch 建立索引时 修改为中文分词器_SevenBerry的博客-CSDN博客_elasticsearch 修改字段分词器

(5条消息) Elasticsearch中analyzer和search_analyzer的区别_chuixue24的博客-CSDN博客

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/空白诗007/article/detail/761491
推荐阅读
相关标签
  

闽ICP备14008679号