To address the issue of inaccurate keypoint detection for turbot in dense scenarios, this paper proposes a fish keypoint detection algorithm named UWO-YOLO based on YOLOv11n. The algorithm incorporates targeted optimizations to the basic architecture, enhancing the model's keypoint perception capability in underwater dense scenarios for turbot. Firstly, a Localization Quality Estimation (LQE) module is introduced into the detection head. This improved method, based on bounding box localization quality assessment, integrates the optimization idea of fusing distribution statistical information and classification scores to improve localization accuracy by optimizing bounding box quality scores. Secondly, a modified C3K2 module enhanced by MambaVision is integrated into the backbone network, where the Bottleneck module in the original structure is replaced with a cascaded "MambaBlock+AttentionBlock" configuration to achieve local-global feature fusion while balancing computational efficiency. Finally, a modified C2PSA module improved by the Efficient Frequency Feed-Forward Network (EDFFN) from the EVSSM model is introduced, which synergistically enhances detection accuracy and inference speed through block-wise attention and frequency-domain enhancement. Experimental results show that compared with the original YOLOv11n, UWO-YOLO achieves a 1.7% improvement in detection precision, a 1.7% increase in mAP@50-95, and a 1.5% rise in mAP@50 recall in dense scenarios. Additionally, it features low resource consumption with a 6.9% reduction in parameter count and a 10.7% decrease in computational complexity. This study provides an efficient technical approach for non-contact accurate measurement of fish length in dense aquatic target scenarios within aquaculture environments.