1. openresty使用笔记

    openresty使用笔记 1. 配置 openresty 环境 通过 docker-compose 安装 openresty docker-compose.yml 配置如下: version: '3.7' services: nginx: image: openresty/openresty container_name: nginx restart: always ports: - "80:80" - "443:443" volumes: - "./openresty_setting.conf:/usr/local/openresty/nginx/conf/nginx.conf:ro" - "/var/log:/var/log" environment: TZ: Asia/Shanghai networks: - nginx-web networks: nginx-web: driver: bridge openresty 配置实例, openresty_setting.conf文件如下: # nginx.conf -- docker-openresty # # This file is installed to: # `/usr/local/openresty/nginx/conf/nginx.conf` # and is the file loaded by nginx at startup, # unless the user specifies otherwise. # # It tracks the upstream OpenResty's `nginx.conf`, but removes the `server` # section and adds this directive: # `include /etc/nginx/conf.d/*.conf;` # # The `docker-openresty` file `nginx.vh.default.conf` is copied to # `/etc/nginx/conf.d/default.conf`. It contains the `server section # of the upstream `nginx.conf`. # # See https://github.com/openresty/docker-openresty/blob/master/README.md#nginx-config-files # user root; worker_processes auto; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } http { ## # Basic Settings ## sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; # server_tokens off; # server_names_hash_bucket_size 64; # server_name_in_redirect off; # include /etc/nginx/mime.types; include /usr/local/openresty/nginx/conf/mime.types; default_type application/octet-stream; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; ## # Gzip Settings ## gzip on; gzip_static on; # gzip_vary on; gzip_proxied any; gzip_comp_level 4; gzip_buffers 16 8k; gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; ## # Virtual Host Configs ## ################################## # demo.com ################################## server { listen 443 ssl http2; server_name demo.com; location / { proxy_pass http://django:6000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_send_timeout 600; proxy_connect_timeout 600; proxy_read_timeout 600; } } server { listen 80; server_name demo.com; rewrite ^(.*)$ https://$host$1 permanent; } } 2. 使用 lua 在请求 header 上写入请求到达时的时间戳 http { server { listen 80; location / { rewrite_by_lua ' ngx.req.set_header("RTIME", ngx.req.start_time()*1000) '; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:8888; } } }

    2019/10/17 技术

  2. mac下 python 报错 CERTIFICATE_VERIFY_FAILED

    mac下 python 报错 CERTIFICATE_VERIFY_FAILED python 下载时, 报错如下: ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)> (caused by URLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)'),)) 多方搜索,得到错误原因urllib and “SSL: CERTIFICATE_VERIFY_FAILED” Error 根据 Craig Glennie分析如下: This isn't a solution to your specific problem, but I'm putting it here because this thread is the top Google result for "SSL: CERTIFICATE_VERIFY_FAILED", and it lead me on a wild goose chase. If you have installed Python 3.6 on OSX and are getting the "SSL: CERTIFICATE_VERIFY_FAILED" error when trying to connect to an https:// site, it's probably because Python 3.6 on OSX has no certificates at all, and can't validate any SSL connections. This is a change for 3.6 on OSX, and requires a post-install step, which installs the certifi package of certificates. This is documented in the ReadMe, which you should find at /Applications/Python\ 3.6/ReadMe.rtf The ReadMe will have you run this post-install script, which just installs certifi: /Applications/Python\ 3.6/Install\ Certificates.command Release notes have some more info: https://www.python.org/downloads/release/python-360/ 解决方法: 步骤 1: 重新安装 certifi: pip3 install certifi 步骤 2: 执行 /Applications/Python\ 3.6/Install\ Certificates.command

    2019/10/15 技术

  3. docker-compose 安装方法

    docker-compose 安装方法 本文提供一种简单的docker-compose安装方法。 compose项目提供了一种通过封装 docker 中 compose 容器的方式,执行 compose 命令的方案。 具体步骤是: 下载 compose 项目中releases提供的run.sh文件, 链接为https://github.com/docker/compose/releases/download/1.25.0-rc2/run.sh 赋予执行权限 sudo chmod +x run.sh 移动到系统目录下并改名为 docker-compose: sudo mv run.sh /usr/local/bin/docker-compose 执行 docker-compose命令: docker-compose ps

    2019/10/14 技术

  4. 系统代理

    系统代理 1. mac 系统 1.1 terminal 设置代理 在 ~/.bash_profile 中添加设置 alias proxytest='curl http://pv.sohu.com/cityjson?ie=utf-8' deproxy () { unset https_proxy unset http_proxy echo "clear proxy!" } acproxy (){ _proxy="http://localhost:8118" export https_proxy=$_proxy export http_proxy=$_proxy echo "using proxy: $_proxy!" } 使用示例: # 设置前 proxytest # 设置代理 acproxy proxytest # 取消代理 deproxy proxytest 1.2 全局设置代理 在 ~/.bash_profile 中添加设置 # global proxy acglobalproxy () { sudo networksetup -setwebproxy "Wi-Fi" 127.0.0.1 8118 sudo networksetup -setsecurewebproxy "Wi-Fi" 127.0.0.1 8118 sudo networksetup -setsocksfirewallproxy "Wi-Fi" 127.0.0.1 1080 } deglobalproxy () { sudo networksetup -setwebproxy "Wi-Fi" "" "" sudo networksetup -setwebproxystate "Wi-Fi" off sudo networksetup -setsecurewebproxy "Wi-Fi" "" "" sudo networksetup -setsecurewebproxystate "Wi-Fi" off sudo networksetup -setsocksfirewallproxy "Wi-Fi" "" "" sudo networksetup -setsocksfirewallproxystate "Wi-Fi" off } 使用示例: # 设置前 proxytest # 设置代理 acglobalproxy proxytest # 取消代理 deglobalproxy proxytest

    2019/10/08 技术

  5. mac 下安装 adb

    mac 下安装 adb 1. 借助 airetest 安装 adb 原理, 使用 python 包 airtest自带的 adb, 实现 adb 安装。 步骤如下: 安装Airtest, 具体为 pip install -U airtest 为 adb 增加执行权限, chmod +x {your_python_path}/site-packages/airtest/core/android/static/adb/mac/adb, 如, {your_python_path}为 /opt/py35/lib/python3.5 设置全局配置: 在~/.bash_profile 中, 添加 export PATH=$PATH:/opt/py35/lib/python3.5/site-packages/airtest/core/android/static/adb/mac adb 使用: source ~/.bash_profile && adb devices 注意,刚方法适用于 linux, 区别在于, linux 下的 adb 位于 {your_python_path}/site-packages/airtest/core/android/static/adb/linux目录下 2. brew 安装(推荐) 使用过程中发现, 方法一安装的 adb 版本过低, 推荐使用 brew 安装: ```shell script brew cask install android-platform-tools ```

    2019/10/07 技术

  6. scrapy项目作为工具库使用

    scrapy项目作为工具库使用 1. 目录树 pysrc - main.py - items.py - ScrapyDemo - scrapy.cfg - ScrapyDemo - spiders - __init__.py - DemoCrawler.py - __init__.py - items.py - pipelines.py - settings.py - utils.py - demo_api.py 2. scrapy项目提供对外接口 在 ScrapyDemo 中 新建文件 demo_api.py, 提供对外访问接口 # coding:utf-8 import json import os import sys root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) if root_dir not in sys.path: sys.path.append(root_dir) from scrapy.crawler import CrawlerProcess from ScrapyDemo.spiders.DemoCrawler import DemoSpider, logger from ScrapyDemo.utils import get_settings, reset_settings def run_demo_spider(query: str, proxy: dict, output_file: str, lang='', max_page: int = None): """ :param proxy: :param output_file: :param query: :param lang: :param max_page: :return: """ # 记录旧的环境变量 old_environ_dict = {} # proxy for key, value in proxy.items(): old_environ_dict[key] = os.environ.get(key) os.environ[key] = value if proxy and ("https_proxy" not in proxy and "HTTPS_PROXY" not in proxy): logger.warning("https_proxy not found in proxy!") try: settings = get_settings() # 修改 settings settings.set("XXX", output_file, priority="project") process = CrawlerProcess(settings=settings) process.crawl(DemoSpider, query=query, lang=lang, max_page=max_page) process.start() finally: # 恢复环境变量 for key, value in old_environ_dict.items(): if value is None: if key in os.environ: del os.environ[key] else: os.environ[key] = value reset_settings() def run_scrapy(cmd_json: str): """ 执行命令 """ # parse args cmd_kwargs = json.loads(cmd_json) # run spider run_demo_spider(**cmd_kwargs) if __name__ == '__main__': run_scrapy(sys.argv[1]) 3. 主项目通过subprocess调用 scrapy 在主项目main.py中, 通过subprocess调用 scrapy 项目 # coding:utf-8 import json import logging import os import subprocess import sys import arrow import shortuuid from items import DemoItem _cur_dir = os.path.dirname(__file__) def _run_scrapy(**kwargs): """ 执行 scrapy 任务""" cur_dir = os.getcwd() try: os.chdir(os.path.join(_cur_dir, "ScrapyDemo")) subprocess.run([sys.executable, "ScrapyDemo/demo_api.py", json.dumps(kwargs)]) finally: os.chdir(cur_dir) def search(query: str, proxy: dict, lang='', max_page: int = None, logger: logging.Logger = None) -> [DemoItem]: """ :param proxy: :param query: :param lang: :param max_page: :param logger: :return: """ logger = logger or logging.getLogger("xxx") # 数据文件 data_dir = "/tmp/data" if not os.path.exists(data_dir): os.makedirs(data_dir) uid = arrow.get().format("YYYYMMDD") + "_" + shortuuid.ShortUUID().random(length=8) output_file = os.path.join(data_dir, "{}.json".format(uid)) # 启动任务 if proxy and ("https_proxy" not in proxy and "HTTPS_PROXY" not in proxy): logger.warning("https_proxy not found in proxy!") _run_scrapy(**dict( query=query, proxy=proxy, lang=lang, max_page=max_page, output_file=output_file, )) # 解析结果 result_list = [] if not os.path.exists(output_file): logger.error("{} not found!".format(output_file)) return result_list result_dict = {} with open(output_file, "r", encoding="utf-8") as f: for line in f: if len(line) > 2: try: raw_item_dict = json.loads(line.strip()) _index = raw_item_dict.pop("index") if _index in result_dict: logger.warning("item index {} exists before!".format(_index)) demo_item = DemoItem() demo_item["id"] = raw_item_dict.get("ID") result_dict[_index] = demo_item except Exception as e: logger.error(e) # 按照 index 排序 logger.info("index list {}".format([_index for _index in sorted(result_dict.keys())])) for _index in sorted(result_dict.keys()): result_list.append(result_dict[_index]) # 删除临时文件 return result_list

    2019/10/06 技术

  7. charles over proxy

    charles over proxy 使用 charles 对安卓应用进行抓包时,会遇到部分应用必须使用代理才能上网的问题。 解决思路是, charles外接代理。原理如下所示: Android App --> charles --> other proxy --> internet 具体操作如下: 安卓中设置 charles 提供的代理,以便抓包 charles 中设置外部代理,设置方法为依次展开Proxy --> External Proxy Settings..., 填写外部代理即可。

    2019/10/05 技术

  8. 使用 markdown 制作 ppt

    使用 markdown 制作 ppt reveal.js提供了一种利用 markdown 生成 ppt 的方法。 可以使用vscode及vscode-reveal插件,搭建书写环境。 1. 环境配置 安装过程: 安装 vscode 打开vscode, 安装插件 vscode-reveal 2. 示例 步骤一、在 vscode 中新建 sample.md, 并写入如下内容: --- theme : "night" transition: "slide" highlightTheme: "monokai" logoImg: "logo.png" slideNumber: false title: "XXX调研报告" --- <style type="text/css"> .reveal p { text-align: left; } .reveal ul { display: block; } .reveal ol { display: block; } </style> # XXX调研报告 --- ## 1. 概述 --- ## 2. 行业现状 示例要点: ppt 基本配置 ppt 样式写在 style 中 步骤二、通过快键键Command + Shift + P 选择 Revealjs: Open presentation in browser, 即可在浏览器中预览ppt。也可以通过选择 Revealjs: Export in PDF, 导出 pdf 文件。

    2019/10/04 技术

  9. docker挂载目录异常

    docker挂载目录异常 场景如下: docker 以数据卷的方式挂载 目录/文件, 如 -v /opt/code:/code。 当宿主的目录/opt/code以如下方式执行数据更新后,容器中的目录 /code 数据全部消失: rm -rf /opt/code/ mkdir -p /opt/code echo -n > /opt/code/new.data 原因是,容器挂载,只认文件inode。当宿主机的目录被删除再重建后,目录inode变化。容器挂载的inode所对应的宿主文件已经消失,故而数据为空。

    2019/10/03 技术

  10. flask 笔记

    flask学习笔记 1. 响应 1.1 展示文本 from flask import make_response @app.route('/') def hello(): response = make_response("文本内容!") response.headers["Content-Type"] = "text/plain;charset=UTF-8" return response 1.2 下载 txt from flask import make_response @app.route('/') def hello(): response = make_response("文本内容!") # response.headers['Content-Type'] = "text/plain" response.headers['Content-Disposition'] = "attachment; filename=download.txt" return response 1.3 图片处理 from io import BytesIO import numpy as np import requests from PIL import Image from flask import Flask, request, make_response @app.route("/", methods=["POST", 'GET']) def search_image(): url = request.args.get("url") or request.form.get("url") if not url: response = make_response("OK", 200) else: try: # load image raw_image_response = requests.get(url=url, timeout=10) # do something image = Image.open(BytesIO(raw_image_response.content)) new_image_obj = image # response new_image_bytes = BytesIO() new_image_obj.save(new_image_bytes, 'JPEG') response = make_response(new_image_bytes.getvalue()) response.headers['Content-Type'] = "image/jpeg" except Exception as e: print(e) response = make_response("error {}".format(e), 200) return response

    2019/07/02 技术